{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 autograd\n",
    "\n",
    "用Tensor训练网络很方便，但从上一小节最后的线性回归例子来看，反向传播过程需要手动实现。这对于像线性回归等较为简单的模型来说，还可以应付，但实际使用中经常出现非常复杂的网络结构，此时如果手动实现反向传播，不仅费时费力，而且容易出错，难以检查。torch.autograd就是为方便用户使用，而专门开发的一套自动求导引擎，它能够根据输入和前向传播过程自动构建计算图，并执行反向传播。\n",
    "\n",
    "计算图(Computation Graph)是现代深度学习框架如PyTorch和TensorFlow等的核心，其为高效自动求导算法——反向传播(Back Propogation)提供了理论支持，了解计算图在实际写程序过程中会有极大的帮助。本节将涉及一些基础的计算图知识，但并不要求读者事先对此有深入的了解。关于计算图的基础知识推荐阅读Christopher Olah的文章[^1]。\n",
    "\n",
    "[^1]: http://colah.github.io/posts/2015-08-Backprop/\n",
    "\n",
    "\n",
    "### 3.2.1 requires_grad\n",
    "PyTorch在autograd模块中实现了计算图的相关功能，autograd中的核心数据结构是Variable。从v0.4版本起，Variable和Tensor合并。我们可以认为需要求导(requires_grad)的tensor即Variable. autograd记录对tensor的操作记录用来构建计算图。\n",
    "\n",
    "Variable提供了大部分tensor支持的函数，但其不支持部分`inplace`函数，因这些函数会修改tensor自身，而在反向传播中，variable需要缓存原来的tensor来计算反向传播梯度。如果想要计算各个Variable的梯度，只需调用根节点variable的`backward`方法，autograd会自动沿着计算图反向传播，计算每一个叶子节点的梯度。\n",
    "\n",
    "`variable.backward(gradient=None, retain_graph=None, create_graph=None)`主要有如下参数：\n",
    "\n",
    "- grad_variables：形状与variable一致，对于`y.backward()`，grad_variables相当于链式法则${dz \\over dx}={dz \\over dy} \\times {dy \\over dx}$中的$\\textbf {dz} \\over \\textbf {dy}$。grad_variables也可以是tensor或序列。\n",
    "- retain_graph：反向传播需要缓存一些中间结果，反向传播之后，这些缓存就被清空，可通过指定这个参数不清空缓存，用来多次反向传播。\n",
    "- create_graph：对反向传播过程再次构建计算图，可通过`backward of backward`实现求高阶导数。\n",
    "\n",
    "上述描述可能比较抽象，如果没有看懂，不用着急，会在本节后半部分详细介绍，下面先看几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import torch as t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9289,  1.4844, -1.1878, -0.7120],\n",
       "        [-0.3095,  1.2505,  0.9202,  0.9983],\n",
       "        [-1.3756,  1.0462,  0.9083, -1.0232]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#在创建tensor的时候指定requires_grad\n",
    "a = t.randn(3,4, requires_grad=True)\n",
    "# 或者\n",
    "a = t.randn(3,4).requires_grad_()\n",
    "# 或者\n",
    "a = t.randn(3,4)\n",
    "a.requires_grad=True\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0.,  0.],\n",
       "        [ 0.,  0.,  0.,  0.]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.zeros(3,4).requires_grad_()\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.9289,  1.4844, -1.1878, -0.7120],\n",
       "        [-0.3095,  1.2505,  0.9202,  0.9983],\n",
       "        [-1.3756,  1.0462,  0.9083, -1.0232]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 也可写成c = a + b\n",
    "c = a.add(b)\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = c.sum()\n",
    "d.backward() # 反向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d # d还是一个requires_grad=True的tensor,对它的操作需要慎重\n",
    "d.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 此处虽然没有指定c需要求导，但c依赖于a，而a需要求导，\n",
    "# 因此c的requires_grad属性会自动设为True\n",
    "a.requires_grad, b.requires_grad, c.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, False)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 由用户创建的variable属于叶子节点，对应的grad_fn是None\n",
    "a.is_leaf, b.is_leaf, c.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# c.grad是None, 因c不是叶子节点，它的梯度是用来计算a的梯度\n",
    "# 所以虽然c.requires_grad = True,但其梯度计算完之后即被释放\n",
    "c.grad is None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算下面这个函数的导函数：\n",
    "$$\n",
    "y = x^2\\bullet e^x\n",
    "$$\n",
    "它的导函数是：\n",
    "$$\n",
    "{dy \\over dx} = 2x\\bullet e^x + x^2 \\bullet e^x\n",
    "$$\n",
    "来看看autograd的计算结果与手动求导计算结果的误差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    '''计算y'''\n",
    "    y = x**2 * t.exp(x)\n",
    "    return y\n",
    "\n",
    "def gradf(x):\n",
    "    '''手动求导函数'''\n",
    "    dx = 2*x*t.exp(x) + x**2*t.exp(x)\n",
    "    return dx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0928,  0.1978,  0.6754,  0.8037],\n",
       "        [ 0.9882,  0.3546,  0.2380,  0.0002],\n",
       "        [ 0.2863,  0.0448,  0.1516,  2.9122]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.randn(3,4, requires_grad = True)\n",
    "y = f(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4146, -0.4610,  2.9016,  3.2831],\n",
       "        [ 3.8102,  1.8614, -0.4536, -0.0244],\n",
       "        [-0.4321,  0.5110, -0.4549,  8.6048]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward(t.ones(y.size())) # gradient形状与y一致\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.4146, -0.4610,  2.9016,  3.2831],\n",
       "        [ 3.8102,  1.8614, -0.4536, -0.0244],\n",
       "        [-0.4321,  0.5110, -0.4549,  8.6048]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# autograd的计算结果与利用公式手动计算的结果一致\n",
    "gradf(x) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.2 计算图\n",
    "\n",
    "PyTorch中`autograd`的底层采用了计算图，计算图是一种特殊的有向无环图（DAG），用于记录算子与变量之间的关系。一般用矩形表示算子，椭圆形表示变量。如表达式$ \\textbf {z = wx + b}$可分解为$\\textbf{y = wx}$和$\\textbf{z = y + b}$，其计算图如图3-3所示，图中`MUL`，`ADD`都是算子，$\\textbf{w}$，$\\textbf{x}$，$\\textbf{b}$即变量。\n",
    "\n",
    "![图3-3:computation graph](imgs/com_graph.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上有向无环图中，$\\textbf{X}$和$\\textbf{b}$是叶子节点（leaf node），这些节点通常由用户自己创建，不依赖于其他变量。$\\textbf{z}$称为根节点，是计算图的最终目标。利用链式法则很容易求得各个叶子节点的梯度。\n",
    "$${\\partial z \\over \\partial b} = 1,\\space {\\partial z \\over \\partial y} = 1\\\\\n",
    "{\\partial y \\over \\partial w }= x,{\\partial y \\over \\partial x}= w\\\\\n",
    "{\\partial z \\over \\partial x}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial x}=1 * w\\\\\n",
    "{\\partial z \\over \\partial w}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial w}=1 * x\\\\\n",
    "$$\n",
    "而有了计算图，上述链式求导即可利用计算图的反向传播自动完成，其过程如图3-4所示。\n",
    "\n",
    "![图3-4：计算图的反向传播](imgs/com_graph_backward.svg)\n",
    "\n",
    "\n",
    "在PyTorch实现中，autograd会随着用户的操作，记录生成当前variable的所有操作，并由此建立一个有向无环图。用户每进行一个操作，相应的计算图就会发生改变。更底层的实现中，图中记录了操作`Function`，每一个变量在图中的位置可通过其`grad_fn`属性在图中的位置推测得到。在反向传播过程中，autograd沿着这个图从当前变量（根节点$\\textbf{z}$）溯源，可以利用链式求导法则计算所有叶子节点的梯度。每一个前向传播操作的函数都有与之对应的反向传播函数用来计算输入的各个variable的梯度，这些函数的函数名通常以`Backward`结尾。下面结合代码学习autograd的实现细节。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = t.ones(1)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "y = w * x # 等价于y=w.mul(x)\n",
    "z = y + b # 等价于z=y.add(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, True)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.requires_grad, b.requires_grad, w.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 虽然未指定y.requires_grad为True，但由于y依赖于需要求导的w\n",
    "# 故而y.requires_grad为True\n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.is_leaf, w.is_leaf, b.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, False)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.is_leaf, z.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AddBackward1 at 0x7f60b09c2630>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# grad_fn可以查看这个variable的反向传播函数，\n",
    "# z是add函数的输出，所以它的反向传播函数是AddBackward\n",
    "z.grad_fn "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((<MulBackward1 at 0x7f60b09c2278>, 0),\n",
       " (<AccumulateGrad at 0x7f60b09c2198>, 0))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# next_functions保存grad_fn的输入，是一个tuple，tuple的元素也是Function\n",
    "# 第一个是y，它是乘法(mul)的输出，所以对应的反向传播函数y.grad_fn是MulBackward\n",
    "# 第二个是b，它是叶子节点，由用户创建，grad_fn为None，但是有\n",
    "z.grad_fn.next_functions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# variable的grad_fn对应着和图中的function相对应\n",
    "z.grad_fn.next_functions[0][0] == y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((<AccumulateGrad at 0x7f60b09c2898>, 0), (None, 0))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一个是w，叶子节点，需要求导，梯度是累加的\n",
    "# 第二个是x，叶子节点，不需要求导，所以为None\n",
    "y.grad_fn.next_functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, None)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 叶子节点的grad_fn是None\n",
    "w.grad_fn,x.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算w的梯度的时候，需要用到x的数值(${\\partial y\\over \\partial w} = x $)，这些数值在前向过程中会保存成buffer，在计算完梯度之后会自动清空。为了能够多次反向传播需要指定`retain_graph`来保留这些buffer。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用retain_graph来保存buffer\n",
    "z.backward(retain_graph=True)\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2.])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 多次反向传播，梯度累加，这也就是w中AccumulateGrad标识的含义\n",
    "z.backward()\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch使用的是动态图，它的计算图在每次前向传播时都是从头开始构建，所以它能够使用Python控制语句（如for、if等）根据需求创建计算图。这点在自然语言处理领域中很有用，它意味着你不需要事先构建所有可能用到的图的路径，图在运行时才构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def abs(x):\n",
    "    if x.data[0]>0: return x\n",
    "    else: return -x\n",
    "x = t.ones(1,requires_grad=True)\n",
    "y = abs(x)\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.])\n"
     ]
    }
   ],
   "source": [
    "x = -1*t.ones(1)\n",
    "x = x.requires_grad_()\n",
    "y = abs(x)\n",
    "y.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.,  0.,  0.,  6.,  3.,  2.])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(x):\n",
    "    result = 1\n",
    "    for ii in x:\n",
    "        if ii.item()>0: result=ii*result\n",
    "    return result\n",
    "x = t.arange(-2,4,requires_grad=True)\n",
    "y = f(x) # y = x[3]*x[4]*x[5]\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "变量的`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都是True。这其实很好理解，对于$ \\textbf{x}\\to \\textbf{y} \\to \\textbf{z}$，x.requires_grad = True，当需要计算$\\partial z \\over \\partial x$时，根据链式法则，$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$，自然也需要求$ \\frac{\\partial z}{\\partial y}$，所以y.requires_grad会被自动标为True. \n",
    "\n",
    "\n",
    "\n",
    "有些时候我们可能不希望autograd对tensor求导。认为求导需要缓存许多中间结构，增加额外的内存/显存开销，那么我们可以关闭自动求导。对于不需要反向传播的情景（如inference，即测试推理时），关闭自动求导可实现一定程度的速度提升，并节省约一半显存，因其不需要分配空间计算梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1, requires_grad=True)\n",
    "w = t.rand(1, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with t.no_grad():\n",
    "    x = t.ones(1)\n",
    "    w = t.rand(1, requires_grad = True)\n",
    "    y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "t.no_grad??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.set_grad_enabled(False)\n",
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.autograd.grad_mode.set_grad_enabled at 0x7f60b01730b8>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 恢复默认配置\n",
    "t.set_grad_enabled(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们想要修改tensor的数值，但是又不希望被autograd记录，那么我么可以对tensor.data进行操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.],\n",
       "        [ 1.,  1.,  1.,  1.]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.ones(3,4,requires_grad=True)\n",
    "b = t.ones(3,4,requires_grad=True)\n",
    "c = a * b\n",
    "\n",
    "a.data # 还是一个tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.data.requires_grad # 但是已经是独立于计算图之外"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = a.data.sigmoid_() # sigmoid_ 是个inplace操作，会修改a自身的值\n",
    "d.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.7311,  0.7311,  0.7311,  0.7311],\n",
       "        [ 0.7311,  0.7311,  0.7311,  0.7311],\n",
       "        [ 0.7311,  0.7311,  0.7311,  0.7311]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们希望对tensor，但是又不希望被记录, 可以使用tensor.data 或者tensor.detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 近似于 tensor=a.data, 但是如果tensor被修改，backward可能会报错\n",
    "tensor = a.detach()\n",
    "tensor.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 统计tensor的一些指标，不希望被记录\n",
    "mean = tensor.mean()\n",
    "std = tensor.std()\n",
    "maximum = tensor.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "tensor[0]=1\n",
    "# 下面会报错：　RuntimeError: one of the variables needed for gradient\n",
    "#             computation has been modified by an inplace operation\n",
    "#　因为 c=a*b, b的梯度取决于a，现在修改了tensor，其实也就是修改了a，梯度不再准确\n",
    "# c.sum().backward() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播过程中非叶子节点的导数计算完之后即被清空。若想查看这些变量的梯度，有两种方法：\n",
    "- 使用autograd.grad函数\n",
    "- 使用hook\n",
    "\n",
    "`autograd.grad`和`hook`方法都是很强大的工具，更详细的用法参考官方api文档，这里举例说明基础的使用。推荐使用`hook`方法，但是在实际使用中应尽量避免修改grad的值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "z = y.sum()\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 0.2709,  0.0473,  0.5052]), tensor([ 1.,  1.,  1.]), None)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 非叶子节点grad计算完之后自动清空，y.grad是None\n",
    "z.backward()\n",
    "(x.grad, w.grad, y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 1.,  1.,  1.]),)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一种方法：使用grad获取中间变量的梯度\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "z = y.sum()\n",
    "# z对y的梯度，隐式调用backward()\n",
    "t.autograd.grad(z, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y的梯度： tensor([ 1.,  1.,  1.])\n"
     ]
    }
   ],
   "source": [
    "# 第二种方法：使用hook\n",
    "# hook是一个函数，输入是梯度，不应该有返回值\n",
    "def variable_hook(grad):\n",
    "    print('y的梯度：',grad)\n",
    "\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# 注册hook\n",
    "hook_handle = y.register_hook(variable_hook)\n",
    "z = y.sum()\n",
    "z.backward()\n",
    "\n",
    "# 除非你每次都要用hook，否则用完之后记得移除hook\n",
    "hook_handle.remove()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后再来看看variable中grad属性和backward函数`grad_variables`参数的含义，这里直接下结论：\n",
    "\n",
    "- variable $\\textbf{x}$的梯度是目标函数${f(x)} $对$\\textbf{x}$的梯度，$\\frac{df(x)}{dx} = (\\frac {df(x)}{dx_0},\\frac {df(x)}{dx_1},...,\\frac {df(x)}{dx_N})$，形状和$\\textbf{x}$一致。\n",
    "- 对于y.backward(grad_variables)中的grad_variables相当于链式求导法则中的$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$中的$\\frac{\\partial z}{\\partial y}$。z是目标函数，一般是一个标量，故而$\\frac{\\partial z}{\\partial y}$的形状与variable $\\textbf{y}$的形状一致。`z.backward()`在一定程度上等价于y.backward(grad_y)。`z.backward()`省略了grad_variables参数，是因为$z$是一个标量，而$\\frac{\\partial z}{\\partial z} = 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2.,  4.,  6.])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,3, requires_grad=True)\n",
    "y = x**2 + x*2\n",
    "z = y.sum()\n",
    "z.backward() # 从z开始反向传播\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 2.,  4.,  6.])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,3, requires_grad=True)\n",
    "y = x**2 + x*2\n",
    "z = y.sum()\n",
    "y_gradient = t.Tensor([1,1,1]) # dz/dy\n",
    "y.backward(y_gradient) #从y开始反向传播\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外值得注意的是，只有对variable的操作才能使用autograd，如果对variable的data直接进行操作，将无法使用反向传播。除了对参数初始化，一般我们不会修改variable.data的值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在PyTorch中计算图的特点可总结如下：\n",
    "\n",
    "- autograd根据用户对variable的操作构建其计算图。对变量的操作抽象为`Function`。\n",
    "- 对于那些不是任何函数(Function)的输出，由用户创建的节点称为叶子节点，叶子节点的`grad_fn`为None。叶子节点中需要求导的variable，具有`AccumulateGrad`标识，因其梯度是累加的。\n",
    "- variable默认是不需要求导的，即`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都为True。\n",
    "- variable的`volatile`属性默认为False，如果某一个variable的`volatile`属性被设为True，那么所有依赖它的节点`volatile`属性都为True。volatile属性为True的节点不会求导，volatile的优先级比`requires_grad`高。\n",
    "- 多次反向传播时，梯度是累加的。反向传播的中间缓存会被清空，为进行多次反向传播需指定`retain_graph`=True来保存这些缓存。\n",
    "- 非叶子节点的梯度计算完之后即被清空，可以使用`autograd.grad`或`hook`技术获取非叶子节点的值。\n",
    "- variable的grad与data形状一致，应避免直接修改variable.data，因为对data的直接操作无法利用autograd进行反向传播\n",
    "- 反向传播函数`backward`的参数`grad_variables`可以看成链式求导的中间结果，如果是标量，可以省略，默认为1\n",
    "- PyTorch采用动态图设计，可以很方便地查看中间层的输出，动态的设计计算图结构。\n",
    "\n",
    "这些知识不懂大多数情况下也不会影响对pytorch的使用，但是掌握这些知识有助于更好的理解pytorch，并有效的避开很多陷阱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.3 扩展autograd\n",
    "\n",
    "\n",
    "目前绝大多数函数都可以使用`autograd`实现反向求导，但如果需要自己写一个复杂的函数，不支持自动反向求导怎么办? 写一个`Function`，实现它的前向传播和反向传播代码，`Function`对应于计算图中的矩形， 它接收参数，计算并返回结果。下面给出一个例子。\n",
    "\n",
    "```python\n",
    "\n",
    "class Mul(Function):\n",
    "                                                            \n",
    "    @staticmethod\n",
    "    def forward(ctx, w, x, b, x_requires_grad = True):\n",
    "        ctx.x_requires_grad = x_requires_grad\n",
    "        ctx.save_for_backward(w,x)\n",
    "        output = w * x + b\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        w,x = ctx.saved_tensors\n",
    "        grad_w = grad_output * x\n",
    "        if ctx.x_requires_grad:\n",
    "            grad_x = grad_output * w\n",
    "        else:\n",
    "            grad_x = None\n",
    "        grad_b = grad_output * 1\n",
    "        return grad_w, grad_x, grad_b, None\n",
    "```\n",
    "\n",
    "分析如下：\n",
    "\n",
    "- 自定义的Function需要继承autograd.Function，没有构造函数`__init__`，forward和backward函数都是静态方法\n",
    "- backward函数的输出和forward函数的输入一一对应，backward函数的输入和forward函数的输出一一对应\n",
    "- backward函数的grad_output参数即t.autograd.backward中的`grad_variables`\n",
    "- 如果某一个输入不需要求导，直接返回None，如forward中的输入参数x_requires_grad显然无法对它求导，直接返回None即可\n",
    "- 反向传播可能需要利用前向传播的某些中间结果，需要进行保存，否则前向传播结束后这些对象即被释放\n",
    "\n",
    "Function的使用利用Function.apply(variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.autograd import Function\n",
    "class MultiplyAdd(Function):\n",
    "                                                            \n",
    "    @staticmethod\n",
    "    def forward(ctx, w, x, b):                              \n",
    "        ctx.save_for_backward(w,x)\n",
    "        output = w * x + b\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):                         \n",
    "        w,x = ctx.saved_tensors\n",
    "        grad_w = grad_output * x\n",
    "        grad_x = grad_output * w\n",
    "        grad_b = grad_output * 1\n",
    "        return grad_w, grad_x, grad_b                       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, tensor([ 1.]), tensor([ 1.]))"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "# 开始前向传播\n",
    "z=MultiplyAdd.apply(w, x, b)\n",
    "# 开始反向传播\n",
    "z.backward()\n",
    "\n",
    "# x不需要求导，中间过程还是会计算它的导数，但随后被清空\n",
    "x.grad, w.grad, b.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 1.]), tensor([ 0.1563]), tensor([ 1.]))"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "#print('开始前向传播')\n",
    "z=MultiplyAdd.apply(w,x,b)\n",
    "#print('开始反向传播')\n",
    "\n",
    "# 调用MultiplyAdd.backward\n",
    "# 输出grad_w, grad_x, grad_b\n",
    "z.grad_fn.apply(t.ones(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之所以forward函数的输入是tensor，而backward函数的输入是variable，是为了实现高阶求导。backward函数的输入输出虽然是variable，但在实际使用时autograd.Function会将输入variable提取为tensor，并将计算结果的tensor封装成variable返回。在backward函数中，之所以也要对variable进行操作，是为了能够计算梯度的梯度（backward of backward）。下面举例说明，有关torch.autograd.grad的更详细使用请参照文档。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 10]),)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.tensor([5], requires_grad=True)\n",
    "y = x ** 2\n",
    "grad_x = t.autograd.grad(y, x, create_graph=True)\n",
    "grad_x # dy/dx = 2 * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([ 2]),)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad_grad_x = t.autograd.grad(grad_x[0],x)\n",
    "grad_grad_x # 二阶导数 d(2x)/dx = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种设计虽然能让`autograd`具有高阶求导功能，但其也限制了Tensor的使用，因autograd中反向传播的函数只能利用当前已经有的Variable操作。这个设计是在`0.2`版本新加入的，为了更好的灵活性，也为了兼容旧版本的代码，PyTorch还提供了另外一种扩展autograd的方法。PyTorch提供了一个装饰器`@once_differentiable`，能够在backward函数中自动将输入的variable提取成tensor，把计算结果的tensor自动封装成variable。有了这个特性我们就能够很方便的使用numpy/scipy中的函数，操作不再局限于variable所支持的操作。但是这种做法正如名字中所暗示的那样只能求导一次，它打断了反向传播图，不再支持高阶求导。\n",
    "\n",
    "\n",
    "上面所描述的都是新式Function，还有个legacy Function，可以带有`__init__`方法，`forward`和`backwad`函数也不需要声明为`@staticmethod`，但随着版本更迭，此类Function将越来越少遇到，在此不做更多介绍。\n",
    "\n",
    "此外在实现了自己的Function之后，还可以使用`gradcheck`函数来检测实现是否正确。`gradcheck`通过数值逼近来计算梯度，可能具有一定的误差，通过控制`eps`的大小可以控制容忍的误差。\n",
    "关于这部份的内容可以参考github上开发者们的讨论[^3]。\n",
    "\n",
    "[^3]: https://github.com/pytorch/pytorch/pull/1016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面举例说明如何利用Function实现sigmoid Function。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Sigmoid(Function):\n",
    "                                                             \n",
    "    @staticmethod\n",
    "    def forward(ctx, x): \n",
    "        output = 1 / (1 + t.exp(-x))\n",
    "        ctx.save_for_backward(output)\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output): \n",
    "        output,  = ctx.saved_tensors\n",
    "        grad_x = output * (1 - output) * grad_output\n",
    "        return grad_x                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 采用数值逼近方式检验计算梯度的公式对不对\n",
    "test_input = t.randn(3,4, requires_grad=True)\n",
    "t.autograd.gradcheck(Sigmoid.apply, (test_input,), eps=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "216 µs ± 42.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n",
      "285 µs ± 42.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n",
      "113 µs ± 46 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "def f_sigmoid(x):\n",
    "    y = Sigmoid.apply(x)\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "def f_naive(x):\n",
    "    y =  1/(1 + t.exp(-x))\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "def f_th(x):\n",
    "    y = t.sigmoid(x)\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "x=t.randn(100, 100, requires_grad=True)\n",
    "%timeit -n 100 f_sigmoid(x)\n",
    "%timeit -n 100 f_naive(x)\n",
    "%timeit -n 100 f_th(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "显然`f_sigmoid`要比单纯利用`autograd`加减和乘方操作实现的函数快不少，因为f_sigmoid的backward优化了反向传播的过程。另外可以看出系统实现的built-in接口(t.sigmoid)更快。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.4 小试牛刀: 用Variable实现线性回归\n",
    "在上一节中讲解了利用tensor实现线性回归，在这一小节中，将讲解如何利用autograd/Variable实现线性回归，以此感受autograd的便捷之处。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置随机数种子，为了在不同人电脑上运行时下面的输出一致\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y = x*2 + 3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size,1) * 5\n",
    "    y = x * 2 + 3 + t.randn(batch_size, 1)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7f6099fa5668>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAEFVJREFUeJzt3V2MHWd9x/Hvv2sDm0C7SbyhsYNr\nkNCqvBScriIgbQRN2w2vMYhKRqWiCNVSS1voxVa4F0SlF7RyL2gvWmQBbWghkAbHRRFkE0EpVRGh\nmzhgh7AlhBCyS/HSsLxlVWz334szG9Zb78s5M3tenvP9SEdndmb2zF/j5/x25plnxpGZSJIG30/1\nugBJUjMMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhdnRzY7t27cp9+/Z1c5OS\nNPDuueee72Tm+GbrdTXQ9+3bx+zsbDc3KUkDLyK+sZX1Nu1yiYgPRMTpiDi1at6lEXFXRHy1er+k\nTrGSpPq20of+98D1a+a9A/hUZj4b+FT1sySphzYN9Mz8LPDYmtk3ADdV0zcBBxquS5LUpk5HuTw9\nM78FUL1f3lxJkqRObPuwxYg4FBGzETG7uLi43ZuTpKHVaaB/OyKuAKjeT6+3YmYezczJzJwcH990\n1I0kqUOdDlv8OPAm4M+r939urCJJKsDxE/McmZljYWmZ3WOjTE9NcGD/nm3d5qaBHhE3Ay8FdkXE\no8CNtIL8loh4C/AI8BvbWaQkDZLjJ+Y5fOwky2fOATC/tMzhYycBtjXUNw30zHzDOouua7gWSSrC\nkZm5J8J8xfKZcxyZmdvWQPdZLpLUsIWl5bbmN8VAl6SG7R4bbWt+Uwx0SWrY9NQEoztHzps3unOE\n6amJbd1uVx/OJUnDYKWfvO9GuUiS2ndg/55tD/C17HKRpEIY6JJUCANdkgphoEtSIQx0SSqEgS5J\nhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklSI\nWoEeEW+LiFMRcX9EvL2poiRJ7es40CPiecDvAFcDLwBeFRHPbqowSVJ76hyh/zzw+cx8PDPPAv8K\nvLaZsiRJ7aoT6KeAayPisoi4CHgF8IxmypIktWtHp7+YmQ9ExF8AdwE/BL4InF27XkQcAg4B7N27\nt9PNSZI2UeuiaGa+PzOvysxrgceAr15gnaOZOZmZk+Pj43U2J0naQMdH6AARcXlmno6IvcDrgBc3\nU5YkqV21Ah34WERcBpwB3pqZ322gJklSB2oFemb+clOFSJLq8U5RSSqEgS5JhTDQJakQBrokFcJA\nl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYWo+ywXaWAdPzHPkZk5FpaW2T02yvTUBAf27+l1WVLH\nDHQNpeMn5jl87CTLZ84BML+0zOFjJwEMdQ0su1w0lI7MzD0R5iuWz5zjyMxcjyqS6jPQNZQWlpbb\nmi8NAgNdQ2n32Ghb86VBYKBrKE1PTTC6c+S8eaM7R5iemuhRRVJ9XhTVUFq58OkoF5XEQNfQOrB/\njwGuotjlIkmFMNAlqRAGuiQVwkCXpEIY6JJUiFqBHhF/FBH3R8SpiLg5Ip7SVGGSpPZ0HOgRsQf4\nQ2AyM58HjAAHmypMktSeul0uO4DRiNgBXAQs1C9JktSJjgM9M+eBvwQeAb4FfC8z72yqMElSe+p0\nuVwC3AA8E9gNXBwRb7zAeociYjYiZhcXFzuvVJK0oTpdLr8KfD0zFzPzDHAMeMnalTLzaGZOZubk\n+Ph4jc1JkjZSJ9AfAV4UERdFRADXAQ80U5YkqV11+tDvBm4F7gVOVp91tKG6JEltqvW0xcy8Ebix\noVokSTV4p6gkFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQtS69V9l\nOH5iniMzcywsLbN7bJTpqQkO7N/T67IktclAH3LHT8xz+NhJls+cA2B+aZnDx04CGOrSgDHQh9yR\nmbknwnzF8plzHJmZM9ClC+jnM1oDfcgtLC23NV8aZv1+RutF0SG3e2y0rfnSMNvojLYfGOhDbnpq\ngtGdI+fNG905wvTURI8qkvpXv5/RGuhD7sD+Pbz7dc9nz9goAewZG+Xdr3t+X5w+Sv2m389o7UMX\nB/bvMcClLZiemjivDx3664zWQJekLVo58HGUiyQVoJ/PaA30Lurn8auSBp+B3iX9Pn5V0uBzlEuX\n9Pv4VUmDz0Dvkn4fvypp8HUc6BExERH3rXp9PyLe3mRxJen38auSBl/HgZ6Zc5n5wsx8IfCLwOPA\nbY1VVhjvyJS03Zq6KHod8LXM/EZDn1ecfh+/KmnwNRXoB4GbL7QgIg4BhwD27t3b0OYGUz+PX5U0\n+GpfFI2IJwGvAf7pQssz82hmTmbm5Pj4eN3NSZLW0cQol5cD92bmtxv4LElSh5oI9DewTneLJKl7\nagV6RFwE/BpwrJlyJEmdqnVRNDMfBy5rqBZJUg3eKSpJhTDQJakQBrokFcLH52qo+Yx6lcRA19Dy\nGfUqjV0uGlo+o16lMdA1tHxGvUpjoGto+Yx6lcZA19DyGfUqjRdFNbR8Rr1KY6BrqPmMepXELhdJ\nKsTAHKF7A4gkbWwgAt0bQCRpcwPR5eINIJK0uYEIdG8AkaTNDUSgewOIJG1uIALdG0AkaXMDcVHU\nG0AkaXMDEejgDSCStJmB6HKRJG3OQJekQhjoklSIWoEeEWMRcWtEfCUiHoiIFzdVmCSpPXUviv4V\ncEdmvj4ingRc1EBNkqQOdBzoEfHTwLXAbwNk5o+BHzdTliSpXXW6XJ4FLAJ/FxEnIuJ9EXHx2pUi\n4lBEzEbE7OLiYo3NSZI2UifQdwBXAX+bmfuBHwHvWLtSZh7NzMnMnBwfH6+xOUnSRuoE+qPAo5l5\nd/XzrbQCXpLUAx0Hemb+F/DNiFh5oMp1wJcbqUqS1La6o1z+APhQNcLlIeDN9UuSJHWiVqBn5n3A\nZEO1SJJq8E5RSSqEgS5JhTDQJakQBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYUw0CWp\nEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmFMNAlqRAGuiQVwkCXpEIY6JJUCANdkgqx\no84vR8TDwA+Ac8DZzJxsoihJUvtqBXrlZZn5nQY+R5JUg10uklSIuoGewJ0RcU9EHGqiIElSZ+p2\nuVyTmQsRcTlwV0R8JTM/u3qFKugPAezdu7fm5iRJ66l1hJ6ZC9X7aeA24OoLrHM0Myczc3J8fLzO\n5iRJG+g40CPi4oh42so08OvAqaYKkyS1p06Xy9OB2yJi5XM+nJl3NFKVJKltHQd6Zj4EvKDBWiRJ\nNThsUZIKYaBLUiEMdEkqhIEuSYUw0CWpEAa6JBXCQJekQhjoklQIA12SCmGgS1IhDHRJKoSBLkmF\nMNAlqRAGuiQVwkCXpEIY6JJUCANdkgphoEtSIQx0SSqEgS5JhTDQJakQBrokFcJAl6RC1A70iBiJ\niBMRcXsTBUmSOtPEEfrbgAca+BxJUg21Aj0irgReCbyvmXIkSZ2qe4T+HuCPgf9toBZJUg0dB3pE\nvAo4nZn3bLLeoYiYjYjZxcXFTjcnSdpEnSP0a4DXRMTDwEeAX4mIf1y7UmYezczJzJwcHx+vsTlJ\n0kY6DvTMPJyZV2bmPuAg8OnMfGNjlUmS2uI4dEkqxI4mPiQzPwN8ponPkiR1ppFA7zfHT8xzZGaO\nhaVldo+NMj01wYH9e3pdliRtq+IC/fiJeQ4fO8nymXMAzC8tc/jYSQBDXVLRiutDPzIz90SYr1g+\nc44jM3M9qkiSuqO4QF9YWm5rviSVorhA3z022tZ8SSpFcYE+PTXB6M6R8+aN7hxhemqiRxVJUncU\nd1F05cKno1wkDZviAh1aoW6ASxo2xXW5SNKwMtAlqRAGuiQVwkCXpEIY6JJUCANdkgoRmdm9jUUs\nAt/Ywqq7gO9sczl1WWMzrLEZ1tiMfq3x5zJz0//yrauBvlURMZuZk72uYyPW2AxrbIY1NmMQatyI\nXS6SVAgDXZIK0a+BfrTXBWyBNTbDGpthjc0YhBrX1Zd96JKk9vXrEbokqU1dDfSI+EBEnI6IU+ss\n/82I+FL1+lxEvGDVsocj4mRE3BcRsz2s8aUR8b2qjvsi4p2rll0fEXMR8WBEvKOHNU6vqu9URJyL\niEurZdu+HyPiGRHxLxHxQETcHxFvu8A6ERF/Xe2rL0XEVauWvSkivlq93tTDGnvaHrdYY0/b4xZr\n7HV7fEpEfCEivljV+KcXWOfJEfHRal/dHRH7Vi07XM2fi4ip7aixMZnZtRdwLXAVcGqd5S8BLqmm\nXw7cvWrZw8CuPqjxpcDtF5g/AnwNeBbwJOCLwHN6UeOadV8NfLqb+xG4Ariqmn4a8J9r9wXwCuCT\nQAAvWvm3Bi4FHqreL6mmL+lRjT1tj1ussaftcSs19kF7DOCp1fRO4G7gRWvW+T3gvdX0QeCj1fRz\nqn33ZOCZ1T4d2c5667y6eoSemZ8FHttg+ecy87vVj58HruxKYefXsGGNG7gaeDAzH8rMHwMfAW5o\ntLhKmzW+Abh5O+pYT2Z+KzPvraZ/ADwArH1A/Q3AB7Pl88BYRFwBTAF3ZeZjVVu4C7i+FzX2uj1u\ncT+upyvtsYMae9EeMzN/WP24s3qtvXh4A3BTNX0rcF1ERDX/I5n5P5n5deBBWvu2L/VzH/pbaB3B\nrUjgzoi4JyIO9aimFS+uTt8+GRHPrebtAb65ap1H2fqXb1tExEW0wvBjq2Z3dT9Wp677aR0Vrbbe\n/ur6ftygxtV62h43qbEv2uNm+7GX7TEiRiLiPuA0rQOGddtjZp4FvgdcRh9+rzfSl/9jUUS8jNYX\n6JdWzb4mMxci4nLgroj4SnWk2m330roN94cR8QrgOPBsWqd1a/V6CNGrgX/PzNVH813bjxHxVFpf\n3rdn5vfXLr7Ar+QG87fFJjWurNPT9rhJjX3RHreyH+lhe8zMc8ALI2IMuC0inpeZq69B9UV7rKvv\njtAj4heA9wE3ZOZ/r8zPzIXq/TRwGz067cnM76+cvmXmJ4CdEbGL1l/uZ6xa9UpgoQclrnaQNae3\n3dqPEbGT1hf8Q5l57AKrrLe/urYft1Bjz9vjZjX2Q3vcyn6s9Kw9rtreEvAZ/n833hP7KyJ2AD9D\nq1uzH7/X6+t2pz2wj/UvOO6l1Uf1kjXzLwaetmr6c8D1ParxZ/nJ+P2rgUdo/RXfQesC3jP5yUWo\n5/aixmr5SoO8uNv7sdofHwTes8E6r+T8i6JfqOZfCnyd1gXRS6rpS3tUY0/b4xZr7Gl73EqNfdAe\nx4GxanoU+DfgVWvWeSvnXxS9pZp+LudfFH2IPr4o2tUul4i4mdZV+V0R8ShwI60LFGTme4F30uq3\n+pvW9QjOZutBOU+ndZoErYb64cy8o0c1vh743Yg4CywDB7P1L382In4fmKE1wuADmXl/j2oEeC1w\nZ2b+aNWvdms/XgP8FnCy6rcE+BNaAblS4ydojXR5EHgceHO17LGI+DPgP6rfe1eef4rezRp73R63\nUmOv2+NWaoTetscrgJsiYoRWr8QtmXl7RLwLmM3MjwPvB/4hIh6k9YfnYFX//RFxC/Bl4Czw1mx1\n3/Ql7xSVpEL0XR+6JKkzBrokFcJAl6RCGOiSVAgDXZIKYaBLUiEMdEkqhIEuSYX4Pw2XbNKCQg8k\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 来看看产生x-y分布是什么样的\n",
    "x, y = get_fake_data()\n",
    "plt.scatter(x.squeeze().numpy(), y.squeeze().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHXNJREFUeJzt3Xl81dWd//HXh03CGtnXEJYIQgDB\n1F1EYkURl7pVrcu0ttRqhU5nbMvv95tfp4/O0paZliCiouDS2sVWazenWAIIQUQCoqgUEkJYEiBs\nCSHrTXLmj3svxBggyd2/eT8fDx6Sy02+p/eBL78995xzzTmHiIgkvg6xHoCIiISHgi4i4hEKuoiI\nRyjoIiIeoaCLiHiEgi4i4hEKuoiIRyjoIiIeoaCLiHhEp2herF+/fi41NTWalxQRSTillT72H68k\nuI+/rqyE+soyO9f3RTXoqamp5ObmRvOSIiIJ5US1jxdyCnnm7V1U+eoBKF7+zdqWfG9Ugy4iIs0r\nq/Lx4vpCluUUcKK6jusuHMi8zDQmDuuN/Xj2tpb8DAVdRCSGyqp8vLB+N8tydlNeXcfnx/tDnj60\nd6t/loIuIhIDZVU+lufsZvl6f8ivHz+QuW0MeZCCLiISRWWVPpat380LObspr6lj5gR/yCcMaXvI\ngxR0EZEoKKv0sSyngBfWF1JeU8cNEwYxNzON8UN6he0aCrqISASVVtayPGf3qZDfmO4P+YWDwxfy\nIAVdRCQCSitrWRYI+cmaOmZNHMTjMyIT8iAFXUQkjI5X+EP+4junQz43M41xgyIX8iAFXUQkDI5X\n1PJ8TgEvri+k0lfPrPTBPJ45JiohD1LQRURCcKyilufXFfDSO4GQTxzM3BlpjB3UM+pjUdBFRNqg\nachvmjiYuZlpXDAw+iEPOmfQzWw5MBsocc6lBx5bANwM1AK7gC8750ojOVARkXhwrKKW5wIhr/LV\nM3vSEObOGENaDEMe1JI79BeBxcDLjR77GzDfOVdnZj8G5gPfDf/wRETiw9GTNTy3bjcvb/CH/OZJ\nQ3g8TkIedM6gO+fWmllqk8feavTlu8Cd4R2WiEh8OHqyhqXrCvj5hj2nQj43cwxjBsRPyIPCMYf+\nFeA3Yfg5IiJx48jJGp5bW8DLG/ZQU1fPzZP9d+TxGPKgkIJuZv8XqANeOctz5gBzAFJSUkK5nIhI\nxB05WcPStf478pq6em6ZPIRvzkhjzIAesR7aObU56Gb2EP43SzOdc+5Mz3POLQWWAmRkZJzxeSIi\nsdQ05LdeNJRvzhjD6P7xH/KgNgXdzG7A/yboNc65yvAOSUQkeg6X17B07S5+/u4eausauC0Q8lEJ\nFPKglixb/BUwHehnZvuB7+Nf1XIe8DczA3jXOfdIBMcpIhJWJeXVLH27gF9sDIR8ylC+eW1ihjyo\nJatc7m3m4WURGIuISMSVlFfz7NsFvNIo5I/PSGNkv+6xHlrItFNURNqFkhPVPBMIeV2DOzW1Egz5\nG+8XsWDFDopLqxiSnMQTM8dy25ShMR516yjoIuJpTUP+hcDUSmqjO/I33i9i/uvbqPLVA1BUWsX8\n1/2fy5xIUVfQRcSTSk5U8/Tbu/jlxr3UNThun+K/Ix/R97NTKwtW7DgV86AqXz0LVuxQ0EVEYuXQ\niWqeXrOLX763l/oGxx1Th/LYtc2HPKi4tKpVj8crBV1EPOFgWTXPvH065HdOHcZj144hpW+3c37v\nkOQkipqJ95DkpEgMNWIUdBFJaAfLqnl6TT6/2rSPhgbHHa0IedATM8d+ag4dIKlzR56YOTYSQ44Y\nBV1EEtKpkL+3jwbnuPNif8iH92l5yIOC8+Ra5SIiEkUHyqp4es0ufh0I+V0Zw3h0ettC3thtU4Ym\nXMCbUtBFJCEUl/pD/ptNwZAP59Hpo0MOuZco6CIS14pLq1iyJp9XN+3HcTrkw85XyJtS0EUkLhWV\nVrFkdT6v5u4D4O6M4Tx67RiGJtjKk2hS0EUkrjQN+Rc/N5xvTFfIW0JBF5G4sP94JUvW7OK3ufsw\nrFUh98I5LOGgoItITO0/XslTq3fxu83+kN/zuRS+MX10izf1eOUclnBQ0EUkJvYdq2TJmnx+m7uf\nDmbce4k/5IN7t25qxSvnsISDgi4iUbXvWCVPrc7nd5v9If/SpSk80oaQB3nlHJZwUNBFJCr2Hatk\n8ap8Xtuynw4djPsvG8Ej14xmUO+uIf1cr5zDEg4KuohE1N6jlSxencfrW4pOhfwb00czsFdoIQ/y\nyjks4aCgi0hE7DlaweJV+bz+fhEdIxDyIK+cwxIO5pyL2sUyMjJcbm5u1K4nItHXOOSdOhj3XZrC\nI9eEP+TN8eryRTPb7JzLONfzdIcuImFReKSCxavz+X0g5A9dnsoj14xiQBRCDlq+CAq6iISo8EgF\nT67K542t/pD/wxWpfH1a9EIepOWLCrqItNHuIxU8uSqPN94vokunDnz5ilTmXDOKAT2jG/IgLV9U\n0EWklQoOn2Rx4I68S6cOPHzVSOZMG03/nufFdFxavqigi0gL7QqE/A9xFvIgLV9sQdDNbDkwGyhx\nzqUHHusD/AZIBQqBu51zxyM3TBGJlV2HT/Jkdh5//KCY8zp15KtXj+JrV4+Km5AHafliC5Ytmtk0\n4CTwcqOg/wQ45pz7kZl9DzjfOffdc11MyxZFEkd+yUmeXJXHnwIhf/DyEXxt2ij69YivkLcHYVu2\n6Jxba2apTR6+FZge+P1LwBrgnEEXkfgXDPkfPyima6eOfG2a/45cIY9/bZ1DH+icOwDgnDtgZgPC\nOCYRiYH8knIWZefzpw+LSercka9PG83Xrh5JX4U8YUT8TVEzmwPMAUhJSYn05UQ8JRo7H/MOlbNo\nVT5/DoT8kWtG87WrR9Gne5ewXkcir61BP2RmgwN354OBkjM90Tm3FFgK/jn0Nl5PpN2J9M7HnYfK\nWZSdx1+2HaCbQu4JbQ36H4GHgB8F/vmHsI1IRIDI7XzceaicrOw83gyE/BvXjOarEQq5V89WiVct\nWbb4K/xvgPYzs/3A9/GH/FUzexjYC9wVyUGKtEfh3vm446D/jvzNj/whf3T6aL561SjOj9Aduc5W\nib6WrHK59wx/lBnmsYhII+Ha+fj3gyf8Id92kB7ndeKx6WN4+KqREQt5kM5WiT7tFBUJiLfpgVB3\nPjYN+eMz/CFP7hadOXKdrRJ9CroI8Tk90Nadj9sP+EP+Px/5Qz53xhi+EsWQB+lslehT0EWI3+mB\n26YMbfH1Pyn2h/yvHx+kZwxDHqSzVaJPQRehbdMD8TJF83FxGYuy81jx8SF/yDPTePjKkfTu1jnq\nY2lMZ6tEn4IuQuunB6I9RdPcfzzSBvYga2Ueb31yiJ5dOzEvM42vxEHIG2vN/8OQ0CnoIrR+eiCa\nUzTN/cfj269upcFBz66d+NZ1aXz5ypH0ToqfkEtsKOginH16oLm742iu4GjuPx7BmOd8d4ZCLqco\n6CIBzU0PnGlqpXdSZ0qrfJ/5GZFYwdHcVBDAyeo6xVw+RUEXOYszTa107dyBpM4dI7qCY9v+MrKy\nd57xz7X8T5rqEOsBiMSzM02hlFb6+M/bJzI0OQkDhiYn8Z+3TwzL/PmH+0t5+MVN3Lw4h02Fx5mV\nPoiunT79r6qW/0lzdIcuchZnW/0S7hUcH+wrJSs7j1V/LyG5W2f++foLeOiKVHp27Rw3SyQlvino\nImcRjc0xW/eVkrVyJ6t3HCa5W2eemDmWBy8fQc+up+fHtfxPWkJBFzmLpqtfeid1xgz+8TdbWbBi\nR0h3yu/vPU5Wdh5rzhLySNOdv7co6CLnELw7Dtdmoi17j5O1Mo+3dx7m/G6d+c4NY3nw8lR6nBfd\nfx3j8fwaCY2CLtJCoW4m2rzHf0e+NhDy794wjgcuHxH1kAfF6/k10nYKukgLtXUzUeOQ9+nehe/e\nMI4HLx9B9xiFPEjH23qPgi7SQq0972XznmMsXJnHurwj9Onehe/dOI4HLot9yIN0vK33aB26SAs9\nMXMsSZ07fuqx5la85BYe44FlG7nj6Q18XHyC+TeOY913ruWRa0bHTcyh5f97JHHEz98ukTh3ruNg\ncwv9d+Q5+Ufo270L/2fWOO6/bATdusTnv2Y63tZ7zDkXtYtlZGS43NzcqF1PJBo2FR5j4cqdrM8/\nSr8eXfj6tNF86bKUuA25JB4z2+ycyzjX8/Q3TqSN3tvtD/k7u/wh/383XciXLh1BUpeO5/5mkQhQ\n0EVaaWPBUbKy8wIhP08hl7ihoEvCiPWuxncLjpK1Mo8NBQq5xCcFXRJCLHc1bth1lKzsnbxbcIz+\nPc/jX2aP575LUhRyiTshBd3M/hH4KuCAbcCXnXPV4RiYSGOx2NW4YddRFq7cycbd/pD//9njue/S\nFLp2VsglPrU56GY2FJgLjHfOVZnZq8A9wIthGpvIKdHa1eicY0NgamXj7mMM6Hke3795PPdeopBL\n/At1yqUTkGRmPqAbUBz6kEQ+K9K7Gp1z/jvy7DzeC4T8X28ezz0KuSSQNgfdOVdkZv8F7AWqgLec\nc2+FbWQijUTqXPJTIV+Zx3uFxxjYSyGXxBXKlMv5wK3ASKAU+K2Z3e+c+0WT580B5gCkpKSEMFRp\nz8K9q9E5xzuBOfJNhccZ1KsrP7hlAl/83HCFXBJWm3eKmtldwA3OuYcDXz8IXOace/RM36OdohJr\nzjnW5/tDnrvHH/JHrx3N3RkKucSvaOwU3QtcZmbd8E+5ZAKqtcQl5xw5+UdYuDKPzYGQ//DWCdyl\nkIuHhDKHvtHMfgdsAeqA94Gl4RqYSGucadORc451eUfIyvaHfHDvrvzwtnTuzhjGeZ0UcvEWHc4l\nCa/ppiOArp068NAVqWwqPMaWvaUM6d2VR68dw10KuSQgHc4l7UZzm46q6xp4dm0BQ3p35d9uS1fI\npV1Q0CXhnW1z0eonpivk0m7oE4skoTnn6NO9S7N/NjQ5STGXdkV36JKQnHOs2XGYhSt3crSiFsN/\noFCQPkpN2iMFXRKKc47VO0rIWpnHB/vLGHZ+Ej+6fSKdOhg/W5mnj1KTdk1Bl4TgnGPV30vIys7j\nw0DIf3zHRG6fOozOHf0zh3dmDI/xKEViS0GXuBYM+cKVeWwrKmN4nyR+csckvjB16KmQi4ifgi5x\nyTlH9nb/Hfm2ojJS+nRTyEXOQUGXuOKcY+X2ErKyd/JR0Ql/yO+cxBemKOQi56KgS1xwzvG3Tw6R\nlZ3Hx8UnGNG3GwvunMRtCrlIiynoElPOOd765BBZK/P45IA/5P9112Ruu2gInRRykVZR0CUmGhr8\nIV+U7Q95at9u/Pddk7lVIRdpMwVdosof8oNkZeez/cAJRvbrrpCLhImCLlERDPnClXn8/WA5I/t1\n56d3T+aWyQq5SLgo6BJRDQ2OFR8fJCvbH/JR/brzsy9O5uZJ8RnyM52rLpIIFHSJiIYGx18/PkjW\nyjx2HPKHfOEXL+LmyUPo2MFiPbxmNT1Xvai0ivmvbwNQ1CUhKOgSVg0Njv/56CCLsgMh79+drHsu\nYvak+A15UHPnqlf56lmwYoeCLglBQZewaGhwvPnRARZl57Hz0ElGJ1DIg850rvrZzlsXiScKuoSk\nvsHx5rYDPLnKH/IxA3qw6N4p3DRxcMKEPGhIchJFzcR7SHJSDEYj0noKurRJMOSLsvPIK0nskAc9\nMXPsZz6bVOeqSyJR0KVV6hscfwmEPL/kJGkDevDkvVOYlcAhDwrOk2uViyQqBV1apL7B8ecPi1mU\nnceuwxWkDejB4vumMCt9MB0SPOSN3TZlqAIuCUtBl7NqGvILBvbgqfumcmP6IE+FXMQLFHRpVn2D\n408fFLNoVR4FhysYO7AnS740lRsmKOQi8UpBl0+pq2/gTx8W8+SqfAoOVzBukEIukihCCrqZJQPP\nA+n4P3T9K865DeEYmJckwnbyUyHPzqfgiD/kT39pKjMVcpGEEeodehbwV+fcnWbWBegWhjF5Srxv\nJ6+rb+CPH/jvyHcHQv7M/VO5frxCLpJo2hx0M+sFTAP+AcA5VwvUhmdY3hGv28nr6hv4w9ZinlyV\nR+HRSi4c3Itn7r+Y68cPVMhFElQod+ijgMPAC2Y2GdgMzHPOVTR+kpnNAeYApKSkhHC5xBRv28nr\n6ht4Y2sxiwMhHz+4F88+cDGfv1AhF0l0oQS9EzAVeNw5t9HMsoDvAf/S+EnOuaXAUoCMjAwXwvUS\nUrxsJ6+rb+D37xexeHU+ewIhX/rAxXx+/EDMFHIRLwgl6PuB/c65jYGvf4c/6NJIrLeT+wIhfyoQ\n8glDFHIRr2pz0J1zB81sn5mNdc7tADKBT8I3NG+I1XbyYMgXr8pn77FK0of24rkHM7juwgEKuYhH\nhbrK5XHglcAKlwLgy6EPyXuiuZ3cV9/A77cU8eTqPPYdqyJ9aC+efzCDTIVcxPNCCrpzbiuQEaax\nSAh89Q28vmU/i1fns+9YFROH9uZfH5rAjHEKuUh7oZ2iCc5X38Brm/0h33+8iknDevODWyZw7ViF\nXKS9UdATVG1dA69t2c9TgZBPHtabH96azvSx/RVykXZKQU8wtXUN/G6zP+RFpVVMHp6skIsIoKAn\njNq6Bn67eR9LVu+iqLSKi4Yn829fSGf6BQq5iPgp6HGuuZD/+xfSuUYhF5EmFPQ4VVNXz29z97Nk\ndT7FZdVMSUnmP26fyLS0fgq5iDRLQY8zNXX1vBoI+YFAyH90xySuVshF5BwU9DhRU1fPq5v2sWTN\nLg6UVTM1JZkfK+Qi0goKeoxV++p5NXcfTwdCfvGI8/nJnZO4aoxCLiKto6DHSLWvnt9s8of84Ilq\nMkacz4I7J3PlmL4KuYi0iYIeZcGQL1mTz6ETNXwu9Xz+++7JXDG6+ZAnwsfXiUh8UNCjpNpXz6/f\n28vTb+86FfKf3X0Rl58h5BD/H18nIvFFQY+wal89v3pvL0+v2UVJeQ2XpPbhZ1+8iMtHnXtqJV4/\nvk5E4pOCHiHVvnp+udF/R364vIZLRvZh4T0tC3lQvH18nYjENwU9zKp99byycS/PBEJ+6cg+LLpn\nCpeP7tvqnxUvH18nIolBQQ+Tqtp6Xtm4h2fXFnC4vIbLRrU95EGx/vg6EUksCnqIgiF/5u0Cjpys\n4fJRfXny3ilcNqrtIQ+K1cfXiUhiUtDbqGnIrxjdl6fum8KlYQh5Y9H8+DoRSWwKeitV1tbxyrt7\neXbtLo6crOXKMX1ZkjmVS0b2ifXQRKSdU9BbqLK2jl+8u4dn3y7gaEUtV43px7zr0vhcqkIuIvFB\nQT+Hyto6fr5hD0vXKuQiEt8U9DNoGvKr0/oxLzONDIVcROKUgt5ERU0dP3/XH/JjgZB/67o0Lh6h\nkItIfFPQAypq6nh5wx6eW+cP+bQL+jMvM42LR5wf66GJiLRIyEE3s45ALlDknJsd+pCi62RNHS9v\nKOS5tQUcr/RxzQX9mXddGlNTFHIRSSzhuEOfB2wHeoXhZ0XNyZo6XnqnkOfX+UM+faz/jnyKQi4i\nCSqkoJvZMOAm4N+Bb4dlRBFWXu07NbVSWunj2rH9uWh4Mq/m7uf2Je9oN6aIJKxQ79AXAt8BeoZh\nLBHVXMjnXXcBhUcqdOa4iHhCm4NuZrOBEufcZjObfpbnzQHmAKSkpLT1cm1WXu3jpXcKeW7dbsqq\nfMwYN4B5mWlMHp4MwGOvbNGZ4yLiCaHcoV8J3GJms4CuQC8z+4Vz7v7GT3LOLQWWAmRkZLgQrtcq\nJ6p9vLS+kOdz/CHPHDeAedelMWlY8qeepzPHRcQr2hx059x8YD5A4A79n5vGPBZOVPt4cb3/zc4T\n1XVcd+EA5mZ+NuRBOnNcRLzCM+vQT1T7eCGnkGU5wZAPZF5mGhOH9T7r9+nMcRHxirAE3Tm3BlgT\njp/VWmVVPl5Yv5vlObs5UV3H58f7Q54+9OwhD9KZ4yLiFQl7hx4M+bKc3ZS3IeSN6cxxEfGChAt6\nWZWP5Tm7Wb7eH/Lrxw9kbhtDLiLiJQkT9LJKH8vW7+aFnN2U19Qxc4I/5BOGKOQiIpAAQS+r9LEs\np4AX1hdSXlPHDRMGMTczjfFDEuqkARGRiIvboJdW1rIsZzcvBkJ+Y7o/5BcOVshFRJoTd0Evrazl\n+XW7efGdQk7W1DFr4iAen6GQi4icS9wE/XhF4I48EPKbJg7m8cwxjBukkIuItETMg368opbncwp4\ncX0hlb56ZqUr5CIibRGzoB+rqOX5dQW89E4g5BMHM3dGGmMHxf3BjSIicSnqQT9WUctzgZBX+eq5\naeJg5mamccFAhVxEJBRRDfrBE9Vc9eNVVPnqmT1pCHNnjCFNIRcRCYuoBv1weQ23XziQxxVyEZGw\ni2rQLxjYk0X3TonmJUVE2o0O0bzYeZ2iejkRkXZFhRUR8QgFXUTEIxR0ERGPUNBFRDxCQRcR8QgF\nXUTEIxR0ERGPiPlpi7HyxvtFLFixg+LSKoYkJ/HEzLH6oGgRSWjtMuhvvF/E/Ne3UeWrB6CotIr5\nr28DUNRFJGG1yymXBSt2nIp5UJWvngUrdsRoRCIioWuXQS8urWrV4yIiiaDNQTez4Wa22sy2m9nH\nZjYvnAOLpCHJSa16XEQkEYRyh14H/JNz7kLgMuAxMxsfnmFF1hMzx5LUueOnHkvq3JEnZo6N0YhE\nRELX5jdFnXMHgAOB35eb2XZgKPBJmMYWMcE3PrXKRUS8xJxzof8Qs1RgLZDunDvR5M/mAHMAUlJS\nLt6zZ0/I1xMRaU/MbLNzLuNczwv5TVEz6wG8BnyracwBnHNLnXMZzrmM/v37h3o5ERE5g5CCbmad\n8cf8Fefc6+EZkoiItEUoq1wMWAZsd879NHxDEhGRtgjlDv1K4AFghpltDfyaFaZxiYhIK4WyyiUH\nsDCORUREQtAud4qKiHiRgi4i4hEKuoiIRyjoIiIeoaCLiHiEgi4i4hEKuoiIRyjoIiIeoaCLiHiE\ngi4i4hEKuoiIRyjoIiIeoaCLiHiEgi4i4hEKuoiIRyjoIiIeoaCLiHiEgi4i4hEKuoiIRyjoIiIe\noaCLiHiEgi4i4hEKuoiIRyjoIiIeEVLQzewGM9thZvlm9r1wDUpERFqvzUE3s47AU8CNwHjgXjMb\nH66BiYhI64Ryh34JkO+cK3DO1QK/Bm4Nz7BERKS1Qgn6UGBfo6/3Bx4TEZEY6BTC91ozj7nPPMls\nDjAn8GWNmX0UwjW9pB9wJNaDiBN6LU7Ta3GaXovTxrbkSaEEfT8wvNHXw4Dipk9yzi0FlgKYWa5z\nLiOEa3qGXovT9FqcptfiNL0Wp5lZbkueF8qUyyYgzcxGmlkX4B7gjyH8PBERCUGb79Cdc3Vm9k1g\nBdARWO6c+zhsIxMRkVYJZcoF59ybwJut+JaloVzPY/RanKbX4jS9FqfptTitRa+FOfeZ9zFFRCQB\naeu/iIhHRCXoOiLgNDNbbmYl7X35ppkNN7PVZrbdzD42s3mxHlOsmFlXM3vPzD4IvBY/iPWYYs3M\nOprZ+2b251iPJZbMrNDMtpnZ1pasdIn4lEvgiICdwOfxL3XcBNzrnPskoheOU2Y2DTgJvOycS4/1\neGLFzAYDg51zW8ysJ7AZuK09/r0wMwO6O+dOmllnIAeY55x7N8ZDixkz+zaQAfRyzs2O9XhixcwK\ngQznXIvW40fjDl1HBDTinFsLHIv1OGLNOXfAObcl8PtyYDvtdKex8zsZ+LJz4Fe7fXPLzIYBNwHP\nx3osiSYaQdcRAXJWZpYKTAE2xnYksROYYtgKlAB/c86129cCWAh8B2iI9UDigAPeMrPNgV33ZxWN\noLfoiABpn8ysB/Aa8C3n3IlYjydWnHP1zrmL8O+4vsTM2uV0nJnNBkqcc5tjPZY4caVzbir+U20f\nC0zZnlE0gt6iIwKk/QnMF78GvOKcez3W44kHzrlSYA1wQ4yHEitXArcE5o5/Dcwws1/Edkix45wr\nDvyzBPg9/insM4pG0HVEgHxG4I3AZcB259xPYz2eWDKz/maWHPh9EnAd8PfYjio2nHPznXPDnHOp\n+Fuxyjl3f4yHFRNm1j2wYAAz6w5cD5x1dVzEg+6cqwOCRwRsB15tz0cEmNmvgA3AWDPbb2YPx3pM\nMXIl8AD+O7CtgV+zYj2oGBkMrDazD/HfAP3NOdeul+sJAAOBHDP7AHgP+Itz7q9n+wbtFBUR8Qjt\nFBUR8QgFXUTEIxR0ERGPUNBFRDxCQRcR8QgFXUTEIxR0ERGPUNBFRDzifwGCq3tAkWXMZQAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.023955821990967 3.0775527954101562\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1,1, requires_grad=True)\n",
    "b = t.zeros(1,1, requires_grad=True)\n",
    "losses = np.zeros(500)\n",
    "\n",
    "lr =0.005 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=32)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y)\n",
    "    loss = 0.5 * (y_pred - y) ** 2\n",
    "    loss = loss.sum()\n",
    "    losses[ii] = loss.item()\n",
    "    \n",
    "    # backward：手动计算梯度\n",
    "    loss.backward()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.data.sub_(lr * w.grad.data)\n",
    "    b.data.sub_(lr * b.grad.data)\n",
    "    \n",
    "    # 梯度清零\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1)\n",
    "        y = x.mm(w.data) + b.data.expand_as(x)\n",
    "        plt.plot(x.numpy(), y.numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=20) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0,5)\n",
    "        plt.ylim(0,13)   \n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print(w.item(), b.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 50)"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXm4HUW19t/Vvfc+U3IyngyQQJgT\nMAwSwqRiAgIC4ohXryIqmusMyqfCvXoFJxBxuA6XCzIqCKigTMoUQAggEEgIhBDGkJHMw8mZ9tD1\n/dFd3dXVVd29x3P2Sf2eJ8/J7t27u7q7etWqt1atIsYYDAaDwdD8WINdAIPBYDDUBmPQDQaDYZhg\nDLrBYDAME4xBNxgMhmGCMegGg8EwTDAG3WAwGIYJmTQ7EdEKAN0ASgCKjLFZRDQWwC0ApgFYAeCj\njLGt9SmmwWAwGJIox0Ofwxg7lDE2y/t8PoD5jLH9AMz3PhsMBoNhkKhGcnk/gOu9/18P4APVF8dg\nMBgMlUJpZooS0RsAtgJgAK5gjF1JRNsYY6OFfbYyxsYofjsPwDwA6OjoOHz69OkVFXR7XwErt/Ri\nn64OvLaxBwAwc/dRFR3LYDAYmolnnnlmE2OsK2m/VBo6gGMZY2uJaAKA+4nopbQFYYxdCeBKAJg1\naxZbuHBh2p+G+Mfz6/DFG5/Fdf9xND56xRMAgIWXnFrRsQwGg6GZIKI30+yXSnJhjK31/m4A8FcA\nswGsJ6LJ3skmA9hQWVHTQUQAgJJjcs8YDAaDikSDTkQdRDSS/x/AiQBeAHAHgLO83c4CcHu9Cume\n2/3rmGRiBoPBoCSN5DIRwF89DzkD4I+MsXuI6GkAfyKiswGsBHBG/YoJePbcGHSDwWDQkGjQGWOv\nAzhEsX0zgOPrUSgVlpFcDAaDIZammSlqJBeDwWCIp/kMujO45TAYDIahShMZdE9yMR66wWAwKGke\ng+79dYyGbjAYDEqax6B7Hrqx5waDwaCmaQy65bnoRnIxGAwGNU1j0MkTXYzkYjAYDGqaxqD7Hrox\n6AaDwaCkaQw6jORiMBgMsTSNQeeSS5p0vwaDwbAr0jQGPZBcBrccBoPBMFRpGoNuJhYZDAZDPE1k\n0N2/RnIxGAwGNU1j0E2Ui8FgMMTTNAadh7kYg24wGAxqUht0IrKJaBER3eV9vo6I3iCixd6/Q+tX\nTFFyqedZDAaDoXlJu0g0AJwDYBmATmHbNxljf6ltkdRYZlDUYDAYYknloRPRFACnAriqvsWJKYP3\n10guBoPBoCat5PJLAN8CIEeB/4iIlhDRL4iopbZFC8M9dJPLxWAwGNQkGnQiOg3ABsbYM9JXFwCY\nDuAIAGMBfFvz+3lEtJCIFm7cuLHigpKZ+m8wGAyxpPHQjwVwOhGtAHAzgLlEdANjbB1zGQBwLYDZ\nqh8zxq5kjM1ijM3q6uqqusDGQTcYDAY1iQadMXYBY2wKY2wagI8BeJAx9kkimgwA5E7h/ACAF+pa\nUCO5GAwGQyzlRLnI3EhEXXDHKxcD+EJtiqTGSC4Gg8EQT1kGnTH2MICHvf/PrUN5tHCD7hiDbjAY\nDEqaZqaokVwMBoMhnqYx6EEc+qAWw2AwGIYszWPQjeRiMBgMsTSRQfckF2PQDQaDQUnzGHTvr5n6\nbzAYDGqaxqBbxkM3GAyGWJrGoPsauhkUNRgMBiXNY9Bh0ucaDAZDHM1j0H0P3Rh0g8FgUNF8Bt14\n6AaDwaCkiQw6l1wGuSAGg8EwRGkag24ZycVgMBhiaRqDzgdFjeRiMBgMaprHoPP0ucZDNxgMBiVN\nZ9CNh24wGAxqmseg+5LLIBfEYDAYhiipDToR2US0iIju8j7vRURPEtErRHQLEeXqV8xgUNRILgaD\nwaCmHA/9HADLhM8/AfALxth+ALYCOLuWBZMx2RYNBoMhnlQGnYimADgVwFXeZwIwF8BfvF2uh7tQ\ndN0w2RYNBoMhnrQe+i8BfAsAT401DsA2xljR+7wawO6qHxLRPCJaSEQLN27cWHlB+cQiY9ANBoNB\nSaJBJ6LTAGxgjD0jblbsqrS0jLErGWOzGGOzurq6KixmcEajuBgMBoOaNB76sQBOJ6IVAG6GK7X8\nEsBoIsp4+0wBsLYuJfTw49AFiz7rh/fjny9X7vUbDAbDcCLRoDPGLmCMTWGMTQPwMQAPMsY+AeAh\nAB/xdjsLwO11KyXUksumnXm8ubmnnqc1GAyGpqGaOPRvA/gGEb0KV1O/ujZFUsM1HiZpLkZTNxgM\nBpdM8i4BjLGHATzs/f91ALNrXyQ1KskFMAbdYDAYOE0zUzSQXMLbTVy6wWAwuDSNQedEJZdBKojB\nYDAMMZrGoOuyLRoP3WAwGFyaxqD7kotkwItmCSODwWAA0EQGPYhyCW+XDbzBYDDsqjSNQddN/TdL\n0hkMBoNL0xh0nYZuPPTmJl90MO38u/Hz+5YPdlEMhqaniQy6a9HlKBfjoTc3/cUSAOCax1YMbkEM\nhmFA0xh0wPXSI4OixqA3NZbJc28w1IzmMuiIxp2bmaLNDe9xmedoMFRPcxl0oqjkYjy7poY/PfMc\nDYbqaSqDbikkF+PZNTfM63GZx2gwVE9TGXQCmZmiwwzHSC4GQ81oKoMOUkwsMoagqTENssFQO5rK\noFsUNeAmyqW5MY/PYKgdadYUbSWip4joOSJaSkQXeduvI6I3iGix9+/QeheWQBEN3cShNzdMvRSt\nwWCogDQLXAwAmMsY20lEWQALiOgf3nffZIz9pX7FC2MRUCjJM0UbdXZDPTCKi8FQOxINOnPjBHd6\nH7Pev0F5DYmMhz7cMBq6wVA7UmnoRGQT0WIAGwDczxh70vvqR0S0hIh+QUQtmt/OI6KFRLRw48aN\nVRWWYAZFhxvm8RkMtSOVQWeMlRhjhwKYAmA2Eb0NwAUApgM4AsBYuItGq357JWNsFmNsVldXV1WF\n5Qm6RGqdnKu7v4B3Xvognl25tabHNaiRJ4oZDIbKKSvKhTG2De4i0SczxtYxlwEA16IBC0aTwqLX\n2kPfvDOPVVv68NqGnck7G6rG2HODoXakiXLpIqLR3v/bAJwA4CUimuxtIwAfAPBCPQvqniu6rdYG\n3UxFbyzmPhsMtSNNlMtkANcTkQ23AfgTY+wuInqQiLrgStuLAXyhjuUEEGTmE6m1QQhmLtb0sAYN\nRkM3GGpHmiiXJQAOU2yfW5cSxaBw0GvvoXuHMwtnNAbjoRsMtaOpZoo2QnLhoosJh2wMxp4bDLWj\nyQx6/QdF+eFMOGRjMFEuBkPtaC6DrthWa2mEH85IAY3BtJsGQ+1oKoOuHBSteZSLJ7kYg94QzH02\nGGpHUxn0RkwscrzoFhPl0hiMPTcYakdzGXTFtlobXuOhNxZznw2G2tFcBr0RkosZFG0oxp4bDLWj\nyQx6dFvRqa2Lbgx6YzEeusFQO5reoNfa7hrJpbGYu2ww1I6mMuiqKBcTh97cDJWG89ZnVmNbb36w\ni1Ex3f0F/OSel1Awo/m7NE1l0Bsz9d/L5TJEDM1wZyhMLHp1w06c9+fncO4tiwe7KBVz2b3LcfnD\nr+Fvi9YMdlEMg0hzGfQGJOfiRxsCdmaXYCh0hPoLJQDAhh0Dg1ySyhkoup65vESjYdeiyQx6dFux\nCovQXyhhe18htM330IeCpdkFGAo5c3jjrapfzUIzl91QO5rLoCu2VWMQ3v+bx3DIRfeFtpkol8Yi\n3uXBNu7DwSgyM8y8S9NUBl05KFqFNrJ8fXdkG7cpQ2Wwbrgj3mdVb2ugWEJfvlTXMnAjSEqXoVlo\n5rIbakWaFYtaiegpInqOiJYS0UXe9r2I6EkieoWIbiGiXL0Ly+151g4qb90GRY2HXjEvrNmOe154\nK9W+Yrsp3/NNOwdwwHfuwbw/LKxl8bRlaGYPnZfd+CG7Nmk89AEAcxljhwA4FMDJRHQUgJ8A+AVj\nbD8AWwGcXb9iumzrdfXuqWPb/W21T87lHde8GRVz2q8X4As3PJNqX/E+F6RJYk++vgUA8Ogrm2pX\nOAXmSRuGC4kG3VsImq+YnPX+MQBzAfzF23493HVF68rmHjdOeFJnq7+t5sm5jIfeUMTHV5QiNPgs\n4NZsY5TBJnbQDUOMJau3Yc22voafN9WbQkQ2ES0GsAHA/QBeA7CNMVb0dlkNYHfNb+cR0UIiWrhx\n48aqCsuN7MjWTGRbNYRiof1B0aoPa0iB6KHLz5J/Vo2d1BJdLPy23jze3NxT13PXCn6HjBsyNDj9\nN4/h2EsebPh5Uxl0xliJMXYogCkAZgOYodpN89srGWOzGGOzurq6Ki+pwIiWrP//Whj0vGC9jeTS\nWMTbLN9zPkjaMM9ZajhO/uWjOO6nDzfq7FXhF93U212asvqyjLFtAB4GcBSA0UTEXeUpANbWtmh6\nRA/dYdXPNhQnY3CjYgx6YxgSHrpm+1s7+ut6XoOh1qSJcukiotHe/9sAnABgGYCHAHzE2+0sALfX\nq5AyI1oyoc/VOumFouChmzj0hiLeZvme+2GMdXbRWY1Oc8O/3sS08+9u6pwwKq5e8Aau+Odrg10M\nQwrSeOiTATxEREsAPA3gfsbYXQC+DeAbRPQqgHEArq5fMcOIHjpQvfE1kkvtKLe3JO4v3/OS91zq\nL7l40o7mRGkTXt345EoAwOqtjR8M4zH0tai1S1Zvw47+YAb1D+56ERf/46UaHNlQbzJJOzDGlgA4\nTLH9dbh6esMZ2ZoNfa7W+OYFD91EuVTHQLG80eQ0Hrpl1cek/+6R13Hlo6/j8k+8HYC+4egrlJC1\nk32fwYySqVUcOmMMp//mMRy2x2j89UvHVl8wQ0NJNOhDkVwm/HJVk88FCHvoJsqlOrr7i8k7CcR6\n6HUeFP3R35d5543fry9fQqfkRKiwvGrZzJ07fs8Xrdw2yCUxVEJTTf3nyC94td602KU2g6LVsXOg\nPIMe9tDD3xUbNCjKn7UqmyeA1KkHuOzRzHXHpI1ubprToEvv3cvru/HUG1sqPl7eDIrWjJ3leuiC\n6quLcqn3lHx5QpNMXyGlQeeyR7UFqoBa3SJjz5ubYWHQz/i/J/DRK56o+HiFJhkU3dabx7Tz78bN\nT60s63cbdvRj5ebeOpUqTH+xvERaog3XxaHXm0LC4Gtvag/dZTAW7eC9i2rPLTeqQ2EBEkN6mtOg\n11hVHWiSQdFVW9zoiRuefLOs383+8Xy866cP1aNIEcTcOipjcNNTK7Fo5VblPlEP3X0u9X4USQO5\n/ak9dC65VF2kiqn23HKj2l8wg0nNRFMZ9P/75OG44szDa94FFycW8fo8FD10zhAuWmzUCgBccNvz\n+OD/Pi7sLxh0jYdeTeO6rTePz12/EFt79LHhvoeuqVdpNXTLjzQZvAdUbb2V8qOhN1+ehGYYXJrK\noJ/8tkk46aBJNT9uPuSheYOiQ9gxGcoGncUYaBXifZYzZ5ZKlQ1Q9xdKKHpG+rrHV+CBZetx7eMr\ntPvz56/r+fWW6aEP5uOptmcpP7O0cpNhaNBUBp2ji3oo1zPiHlU4ysX9OxRH+5shX7d419LcQnEX\nXRx6uSmSP3z54/jNQ69KJ9Ifww9b1dzf/rI19FS714Vq663cePYMEQ+9UHJw8d+XYXtvIXnnXZim\nNOj6GX3lVeaMFzg81KJcNuzoj5UIhjJxuVkS99fEoZf7KNZt7/fHG9KQT9DQ00a5WDUamKwE/k6U\nqlwkWm48ewaGhod+95J1uOKR13HJPcsGuyhDmuY06BpXaqDMCAvbc9HzQywOffaP5+OwH9yv/X7o\n9R0C4qJWVIQmFmni0Mv1Ooslx68LaQbQA8lFXa7UUS7eAQZzULRaDz0qucR76H9dtBrTzr8b3f31\n9Zz5MyrXadvVaE6DrusaCyPyv5r/Cq577I3Y42S4QS9GwxaHYpRLMxBnoFWEBlEjHroX5VLmsyg5\nLBKdEXeEvGJQVAyZjPPQ//fhVzHt/LuRLzrC9PvG1x1/ML/Keiv/PMlD/82DrrS1vs6ZKYN1Xw1x\nNKdB12wXPfSf3/8yLrzzxdjj2N7apKE49CEcttgUGnqMgU7aXzZGvoZerofusLJ6a2KDni86mHb+\n3bheGESNM5KX3rPc/V3JEWaKllXcmuCH21Yd5RL+fdJ95LvXfzZvY87T7DSnQdc803ITQyk99CEc\ntlhJkdJmCqwV4n2T76HKMKbJh15uzvuSwzBQRvx0MLGIfInhh3cHWm2qaB3G/FwugzGgzu9VtZOx\ndM8gaf8L73wR086/G5+65qnUcfvlwOuJ1ZQWq3E05e3JaJ4qr0hpjRhv7cMzRYeuh84rdTnGrdFh\nZyENXTYOinLHhTmKxintJTPGUHSYP2M1jUPna+iafdPIGI7DAg99EOqOP/ZTteSifwYq+HvyyMsb\n/b9Pr6g8DYeOoFhD30MfzHkITWnQx3TklNtP/dUC9BdK2LyzvAiRULJFFv47lFA1MowxPPrKRu2L\nXA9vKY44A60qf1wDIH5O6/Xyn8geetzPRQ1ddRvT9NYcJkSaDIJB5+estkOmzXipsaONulZer+qU\nSbmmDKbtSLNi0VQieoiIlhHRUiI6x9t+IRGtIaLF3r9T6l9cl3Eagw642f427Rwo63hhmcD9OxTj\n0FWG5YFlG3Dm1U/hGs0A8KB66FJx1QY9nYeeVgIregOpvoee4jei5KYqo85IymkL+MSietSd//jD\nwthVg3ixq5UKy8142aj3ZChp6ItWbo2N6hlMuTaNh14EcB5jbAbctUS/TEQHet/9gjF2qPfv73Ur\npcS4EXqDDgAbUxp0VYhivQZF+wulssMqZVRF2tHnVqwX1mxX/ibttPVaEY5ySSO5CN/H6LdpZ+7y\n35SjoecFDV31MupeUHFlIsaY7z3Wuu6UHIZ7l66PXTWI3+tilVOcdatG6TxjVc+w1rmWgKBeDbY9\n78uX8MH/fRxfvOFZ5feMMVz+8OAt15do0Blj6xhjz3r/74a7nuju9S5YHPKaoiIOY77k0pGzY4+j\n8mpqFf4lM/279+Doix+s6hgqQzGqzV14YXuf2mPoKzR2pp9YwnIHRWW7WaxAcuG/KSfrY7KHrj73\ntY+tCJWPEvavlFVbkjNllnxHpLpz6Wbr6nLFqzR2+bmv3daH9/9mgdbpSAM/4mB76LyuLFmtXgDk\nxXU78LP7X25kkUKUpaET0TS4y9E96W36ChEtIaJriGiM5jfziGghES3cuHFjVYUVjqn9znGCuOGW\nbLxBV81E9AdF69Bt2lLl7E+VQWzzGq1tOoOeH7wolzQRE/EeenTCVxJ8pmQ5YweFkIauMOhaDz0w\ntK6GXp8FLl56qxsA0DWyRbuPX5drPCjKj1uOhy4HJbz01g48t3o7zvi/ylNcl3NZtegN6+B1Qbcs\nYrmRdrUmtUEnohEAbgVwLmNsB4DLAewD4FAA6wD8TPU7xtiVjLFZjLFZXV1dNShyPCXG/FY0qTV3\nFC9BMPW/PuWrBl5MlRHUeej1qtg64maKlq2hC7MC0xoq7jEOFJ2Q/MNiphaF1pRVPHfduUNjL079\nPfQpY9q0+9QsDl3TSypHQ5dTKXAJJm0KBRXBoGiyh/656xfiwjuWVnyuOLikxcvRly/hH8+v878f\n7PzxqQw6EWXhGvMbGWO3AQBjbD1jrMQYcwD8DoO0YLSM4zDfQ8gkDImrNPRaDS7VA9XLw7fpkhY1\napEIDlPcS45SQxf+H9HcHf2xdPDfMOZN9kkTtii03qoy6uqC3Hj5g6I1vucFJ7k+84ao6myLmmdg\nlyG5yF5qLe4HfwRpolzWbuvDik31WdCFOxm8HBfctgRfvPFZvPTWDgCD7wimiXIhAFcDWMYY+7mw\nfbKw2wcBvFD74ul59Ftz8KuPHxbZ7ggeup3w9FW6Y9o49GsWvIHlXle4USg16AQPXfzNjU++iYeX\nb6hP4TzSDnJy0ka5pDUK4qDggGLCmIogDp0So1z+86/P49ybF0XL7gSDonHOQKHk4IZ/vVnWhK80\nvZOgLtdWcklyCFRl4/dz/rL1+NuiNaHnWmn5gnVfk/cdKDroHqhPbpmg1+gW5BlvsRY+N2aw56+k\n8dCPBXAmgLlSiOKlRPQ8ES0BMAfA1+tZUJmpY9uxb9cI//NpB7vtS8lhidnzOPzdr2RQ9Pt3vYhT\nfvVoGSWunrgBO92LJ75M//XXF/Dpa5+uT+E84jR0lZwRN/U/7KGHv+sZKGJjdzSaSfzNhbcvxYoU\nS+/lvZeUoO4yi+desakHb2zqUZQvMDZxRvDO59biO397Ab8V0vuu3tqLi+5cqpXHuO2Pa5T4vav1\n1H8+jqE7qup8/DrOvn4hzr1lceg+VZqOlx8hjeQyUHSwo68+wQAFX3JxP6/f4dZBStGYNwJ9uIgH\nY2wB1OG8DQtT1CF64EdMG4u7lqyDw4IudFJrqZp5yf8f92Aale+lZ6CIloyFjG2FyiTqwUkVqNEe\nQ2iAOaLHRi26E+OFx8Whv+/XC/D6ph6suORU7W9uW7QmVZlDUS7SeWyLIg0Lz/gXaoxYupmibd5A\n/YJXNuHcE/YHAPzhiTdx7WMrsPvoNnzunXtHflNKUSf9faqVXDQeunKw2GHKRkaWXDYIibu6+4vo\nbM3GlsFxGD5x1ZPY0N2Pe859F7K2JXjoabJnlkID6rWkUApr6LzuDJUcUE05U5QjGvSWTGD0+E1O\n8lb8bqr3d0N3v5+lL+63jXpoB33vXpz35+f8z+qXKv4YjTfoeglFVX5xi5wZVXwp5et43fOSZdJe\n70MvBdKTGOUi/74ta4euw2FBwyTnrfFzucSUgX/1vBDCxwc7/y4MroV+43vf+uvhx612zES2g3Gr\nRulko7y0XUySlybN7va+Ap54fTNe29iD7n7X0+anTyu57Ogv1mWAUtbQOfy2D/aExOFj0LOBhpXG\nQ2cs8C74brN/NB8/+rublCmugW/kQ7t98drgvIoyJZWl0YOiooWORrlEd5cjRUQqyeVS1Fg9eetn\nrgukJ1VyNk5r1g6Vy3GYf46S1LvgHnqc4eVyRHhhcvevLlVtmpWbVBFblaDT0FWH1b1fcZO6dvYn\nSyEF4eXjkTH8upLsOWPu+19yWNlRNcWSg5/e+1LsqkhB4x8uSa1y6VRLcxt04abmbLcrG/LQHQbG\nGNZsi65eE4pQUGnTMRak2t5cX76E9//2sbInWgQSkViWBFlpED30OE082F/4PibMMa3nXUmPRFzg\nQv59a9aKSi5cV5blJa6jxpSBG3IxYiUpTTDfHtc4+7lcGEPJYfjrotUV3QvtM1AcSlce2UMX6U5h\n0MVy85nOSeGTYpn4JZSro9+z9C389qHX8OO/61dF8sshWU43B38JNz21sqxz1pqmNujiTc1xycUJ\nXlDHYfj9E2/i2EsexNK1YeMpv6Ry9yzuZUjjofcXSnh1gzoKZtGqrXhu1Tb84K74fO0yOh0zjkZ7\n6CEDncKgy/lQRIolhqwdP1lHfm6VTH3nBogh+mzbsnYkP01RIUO4US5BLpf1O/px4R1L/cWqOQOe\n15ixSfhtwsAjN9Yx1yZGufz+iRX4+i3P4U8LV2n316FL16BMiaAz6DFBCTtSSC5iL4tPECslNHoc\nsedT7ipKvNxxczdEDT089uauwfDAsvpGkSXR1AZdlFxsIRc1fzBFh+HJNzYDQCQuVY49V9VNXYVN\n4/lccNvzOOHnjyi7b8Hak4mHSTyvqnHpGSjiD0+sAGOs4aPu4QHb8HdJMd4qjz4rDQjL1CLmmdeX\nh5dvxJekHB0tWQuyVs4HRUvSdjGXy3/e9jyue3wFFry6SVleMQV00rJqgUFnuPO5tXhre3R1IEfY\nh6+n2jNQfqRH5JnFaOg6Z4EbRFXU8M4UZRKPu623gB39hdTLEYqNSZrGQySIddf3AgINPTpYvqHO\nqzalYdgYdP4QbvjXm35oVKxsIhkSlWen+73K0DPGcNWjr/uZHp983W1IdirCtKwKp4irer8qA/aD\nu17Ed29figWvbtJqyjrOvPpJzL3s4bJ+oyqj+/80HnrwHFURFrznpevF3/CvNyO/KRdRInhLeilt\nxYvrL43HEIo9F2eKFrzfyKUZUMyRCCZDxUsu+ZKDr960CKcqwmX9maIO8/P3tCXkMlKhizRSOjya\n8nKj2pKJnj+N5CL2aj559ZM4+ML7/G1JEqLoXe9IcS4R/8gxqg63E0Ryg17WqepGcxt0ihr0vzyz\nGo++4npFoSn90BsXx9MdZXTensrQL127Az+8exnO+5MblcJzPagqYJoJKCqUWr80YAcAG7z47HzR\nKfscj76ySRtBkoY4CUWdyTAwbqoY6FyCh/7Du5dh2bod2nMG5dKXOU4isCySGv/AS2OMCSGlUM4U\nlW2DqjsfZzTF4/FyblbkBOLttsOYP7jakUuMSo4QbYSjyzPK5Z65+yj86INv87fzRmuv8R2R46dJ\n56xqlPnEuSTDKT7Lz1z7dCjePwk/o2OMRS8IcxYqGeOpN81t0EOSS/QhxHlrokPusHRZ44LfRrfz\nSswrHi+P6rj+YsJyeUvuepYycWGYYhn5uYKBG/XMx3riSA2liE5Dt8iL95YNRimQXOKu4/rHV+Bj\nV7qJn2o9ZqDy0PmgaMlh/uAmH4AHXFnhre3RgXggiAAJx9+723RhgPz8cdKMKLnwZfRsizBQLOGi\nO5eGZhJv3jmgTV4WN1NUvrX8nGcdMw1nHD7V3+7HZiuO35diYpGqV8kTlCXVZ1mCu+y+5Ynn48SF\nRu4cKGLxqm1ayWWwc7hwmtqgWwrJRYbf54eXbwzlBpfjpUuKSqSrO6rtcr5mW+GtcYKsfOHtvZqX\nbFtfPnQsnRcsZ9yziRqfy0X4f9rkXATyDGf4u/5iyU+V/Kv5r+CBF9crz3nz06vwr9e3eOdQG0W5\nhzayNZ33KjeKDmPCoGgQreIw5j/Pqxe8gZfX7wQQDW8bEMZ3OPx4OnnMN+gx+rCos3MPveg4+Nui\nNbj2sRW47N7AsB3+wwfwqaufij2X6rPO2NsW/MFrIByUINKes1OFEqrkT94LS+pxyr2t9oSMqyK8\njqgsyZdufBYf+O1j/n2PauipT1NXmtqgi5KLLm/LP154C4ArxXz39iDdjOgN8nUoZcqRXPgWXgpL\neNHl1pt/1G2X6fVeUJUXIJaDXv+uAAAgAElEQVSRvwj8r23RIIQtimULf6fT0C1yI5bEl7VQclAo\nMbS3uC/kfS+ux99fUE+8EUk7ZlByGD7/zr0SJ6rYRNKMUB4ax+CIg7aOPkpFhEsuYh0KelYaD13Q\n0Dk7pNw9YrbFXn9tXaY99lOadT/jPHS5+vmJuywLRISPHD7Fu0Z1j3JES0YpuWzvLYQMsepdLAoN\nVhyypNUes3aCTJyHvsjL2cJ7NvIktMGe8s9pboMe8tCT939D0IblkDOl96ipPHFJsnhPgTc2hZIT\njRzwNVPpC52GyvSVWZU7hb+7ROqXo79QqpuhD2dblLw9jYZuEUWkDf7iiIuZJBlr3XNUUSg5yNpW\nYkZOy4pGswDufXUY88MPHU1EUVRDj056S5JU+LMSDbosLfBz7+wv+rKG6n4lpsMQE9VJY0u6Hhev\n65edcQhm7zXWl44ch4Xe0RGtGeUKWod8/z7M+8NC/3PccxaL//DyDZh2/t14dUM37lqyFotWbo3c\nl/YyBob5oZW9fe9LMWwxVC+GiIs+fAx6mavHRjX09FEuqpdCXGhYLE+hFDUy/tRx6ZRJmr2qnote\nsBza5TjqY07/7j249N702mI5xIUh6lYsIopKG7xrLr6QSRkKCyVHLzGJXrbjhh5mbSsxI2eka809\n3pJn0HmWPUVPTAXX0EXNvZikofs9uuhx/H28cr21oz8kucgDfEn58aNpDtJ46ME2myjUW8gJX+o8\ndMCVRDlxcwnEOsRTJSxcsRUX//0lXP/4iojk0laO5BLjofOz8vtuWbKHnvo0daW5DboouZS5NJUs\nuTz4UnRCgDYOXfHi8gfNXyBej4veNOTwcb2/KTxYcTs3AK9t7MHZ3tR12UNfunY7nnnT7R4WHL2B\n++ui1crt1SLLEyKqsjDGYFnkykPCj/u9lZY6BA89KV67v1DSeqAhOcd7ALmMhaw85U9CLhc/fMFx\ne15cO3YlmOjv5WopGlR+rJLQSKhQjQvIhpkfq7/g+KGXqugdlYe8emuvr1GLjVLRcRC3apQouXDE\nZGYOC8JOAde4yudXvWOxM2KFMvB3jcF99kWHRTz0jnIkl4hwGqXfj7GPjq0MBZraoKcZFBUR9xAr\n0gPLNuC/b4+ucCIb2E07B/CnhavUOaC5dyUNiuZLjiK+Wu2BJE1kEivQfK8BCmvoDKf+aoH/+fZF\na7QL1o5ui19ou1JCGrp03WoP3b1lsuTCPXQx9E70YFVSyaHfv18bcimemjcMWZtg2/H1RhXlAgQe\nuh9D78SvisQRDY4stRR0Grrivg0UHfzhiRW+IS45zE9Q5zcQKrlNYeTf8ZOH8N7/edS/DvG8oVWj\nNA6I6KFbFgUhlA5CBr09Z6NXWuNWlSYgVnIRrkl85fsLpVDaD/GcaeGHVnro3rXy5H1y7vxy5qzU\nk6Y26CIJjlaENC1qX74UCu/64g3P4Ft/WYKVikV7ucckD4oWStEIGp2Grq8U+u9DlUo6z9+ExF4y\no9vjU5hWGoYVO/NTOZjsTpmX4719yaVFLbnopJKXNYuOiPeJR4tkbSuxSx6JcvG9aTfGX5zJqnp3\no5JH1KAH0Uua2cAKO9/dX8R3b1+Kj3rrdDqMYXdpiTo57QCAxNmM8mIUcbJCyR98Fzx0CodQipJL\ney6qoasSecWlOFBm7GRuQ1UsRT108fyJ+HHoiq+8v+IsWDlsUfX8G+25p1mxaCoRPUREy4hoKRGd\n420fS0T3E9Er3l/lItGNIkkLBYD13f349LVPYfPOgVSDZ3N/9k9M/+49/mfelVVpnb7k4hWDe5BF\npYfODXr4GElRNaq6EY5DT5/HZEx7vIdei3DHdGGLrrcT8dC9F39ETj0oqhvM1HXUxAaKP79cJtmg\nq6JcAKDguBKLPyjqqDX0OMmFP1fxXqvqlsoovL5xZ+g7x2HYfXTYoOelBv7NzT344P8+HjmWv7+0\nDmvJYZKGLj9P968cbSYmCmsRJZdcVHJRafpx0pp6HMnxGx/ZQy+UUY99wSXGlPgeOuRJfWqJq9Ha\neprmqwjgPMbYDABHAfgyER0I4HwA8xlj+wGY730eNNJILqu29OHh5Rtx1YI3KrrR3F6qDGteGP0W\n/xaUGrraQ9fZY5XkIn+n+17HiIQ4bNmwqFYHUiF65W/t6MeB/30Pnlu1TVu+0MQi4ZRcqxTDzsTu\neUbjeenqgXhqfpysbaE1xqAfvfc4bZRLyZdcgpmiqtsv1xXRI+U9KtEjFe/7nc+txQtrtivv26sb\nXIM+obPVPQZjkZmhsoe+Zmt4spNsgLb05CMSnvj53674F65/fIXwfTSNgeUNit7y9Eps7B5QSC6y\nQVd56O45VXMFVJKLuIYB71Hf/bV3IGNRKHY/Cf6sVHWIf8ePzxCtF6rGiYctN2riUaJBZ4ytY4w9\n6/2/G8AyALsDeD+A673drgfwgXoVMg3iQ7j0Iwfj/PdO1+67cnNvVV0h8f167NVNKJSciIfOK3le\ninIJxbzLHnpCVE1Scq5yvOok4y96SQtXbMERP3oAdy3RSzgc8bALXtmE3nwJVy14Q3tOx4EX5RJu\n4Pq5hy5ILmIPpFwPvRTy0N3/52xLm+/kn998N26ad5RroBQaujsoypAV8tAkLV8HSJILP5Zwr8Ve\nyFdvWoTTfr1Aed9e8zz0rhEt7rEcIJsJv85Fh4V0fTkmW06UtblnQPI6wx768vXd+N4dS3HHc2tx\n+A/uV+alsS1CoeTg27c+DyCsobfl7EiUi8qg80ZNtbJR+F645+0TMjLyiT8HTByJI/cemxgZJRJM\n/Vd8h7CG7kTkKKYcD3AYw14X/B0fiOkZ1ZKylGcimgbgMABPApjIGFsHuEYfwATNb+YR0UIiWrhx\n40bVLjVBrFTv3G88Joxs0e67cktvRVPi/e6t8JJ+4qon8csHXhY0dB7lopZcHEEnlUuglVx8jVVh\n0AUDIGuWcSgrn6brv3StO/D2Ly/hWBzivRHvgfwdh2voukHRdnFQtChILprBTN09fH71dj/7YCHk\nocd7+rZFKDgO/u+fr6G7v+A3WMUSQ0mQXNJqqOJ95Q2UWOa7n18Xmi+huybuoXe2ZbGxewCbewaw\n2+hW7bmAaCMop5fd0pMPh0YWw1EugNtgXnTHUmzuyWPzTncGsxw+zNfZBMIadlvWRr7o9li39uRx\nzYI3lCkI+PWOUESoqOoQP0axxLC9r4ARLRlkbAtZ20pl0Pn6tIHkovfQxYlh8vhCnOTCe6n1JrVB\nJ6IRAG4FcC5jbEfS/hzG2JWMsVmMsVldXV2VlDEVYl21iWI19Z6BYtke+vod/VjnpS2Vteo3N/f6\nngZvyUXJRV43Ux4UfeTljdgZUyYxrleEMRbx0HOZlAZdUflEIy++CEEyseTjirvwovFjie9WsHYr\n/EFR8Vr6FBOLCiEPXX2dukRbz6/Zji/e+Gxon6xNWg2d1x+bCKu29OGSf7yEC+94MfDQS67eLOaa\nUScfC28rOixYUNgJtnG+87cXMOeyh5VauwjPJFh0HDz00gYwBpx00KTQPn35El7bEDQOcg+O53fp\n9KSNLT350Lkef22Tn8KAM3VMu1/HxJwxHJso5PnLkgv/3bduXYLv3/UinnojOmOV695KyUUMW/RO\n63vozDXoo9pczz5rW5FxBBXv+/UCHPGjB0L1+7J7l+NixUIXwTyC6ExRVW9jyA2KAgARZeEa8xsZ\nY7d5m9cT0WTv+8kABjWzuyUNzMRp6gNFp+xVh4788Xz//6KnCLiVlhsJ/p0ouciLHYsGfc22Pnzq\nmqdw3p8WJ3rosrPhXofYWDhVGXTRiIvXyD2WNHVT7EV0+x5xVCvmxeYTi2xJ2uCDZ7qJRToPnTdK\n3zl1RuS7RV58/s1Pu6vKZDN6DZ0/PzE0dt32vlBIoDuxSJ2Xx79O6TYXS44/UBh46NFnIS7V9tqG\nnZHvOQMFB69s6EZLxsLM3UeFvrv56VW45rE3/M/yeXjWxpGetLFpZ1hD57KJyO6j2/x7z1PhZiTJ\nRUSsj9xbL3ieNIBQ0jC/nN4zVI3ziOXjZ+oXJmvt6Cug0zPouZQeOg91FfMx/eahV3HFI6/7+/Cz\n8rGdZet24Amhx6r10Ida2CK5b/PVAJYxxn4ufHUHgLO8/58F4PbaFy89YkXKWPFTulWx4eUgxwu3\nZGy/dR6QBkfliUVFQZd0HKDXM3qvbexJjHKRW/uegWLYQy+VIbkoDboguQjXyO9tmoEdsYzd3ssa\neOhRLZox9wWyrXAiMd6N7oiZ+j9+RAumjWsPbeMe1GeP3QsnzJgY+o638Tf8yzXouZiwRTnJGuBq\nzvzy+HP10+dqPHRVBkmeJ9z30BVepJg3XJUulzNQLKEn7yYxS5pcJ9s2Lpnw9Xj78skLK2ds8nsl\nP/VmG4vOk+xIifXRtoOGjJdVlayr6HvoCg1dUTxfcnG4h+7WmaxNIYN+0i8ewWHfvy/2+oBwqKkv\nCXHJRRjUvvSeYLa14+g89MTT1ZQ0b/+xAM4EMJeIFnv/TgFwCYD3ENErAN7jfR40QpXKik8FIOYJ\nTzPBVI4WkF/Alozld5EL/kCR+50c5eJKLsHxxCMlTf2XW/uegVJkIkhaD13luRQUkssfnliBdd6a\nrGm6j2IR+QDV469txh+eWBF6GcVeikWE8SNaQpE0fYUSbItCYW9i+RzG8K79xmPqWMmgF0t+KoGs\n5MVbFoWeZTZmUJQbHFHZ2dlfFCQXd+3KULZFhTOoyiHOr+lDlz+OfFE9m/eVGK9cpL/goHegiI6W\nTGL6C1kq3NLj3m/R001yaPNFJ+I0iL0l2Z8Q6yMfQJ6/bIPv3apmrvL7odLQVQ0OP4bjRCUXMcpl\n+fpubI1ZADpYtzTYxsOWuZS6fL16noNqUhPf3kgS58UyxhZAPxf2+NoWp3xOmDERb99zdOjFy1hW\nrLeSF6SKrGXFLmoLREfiZWOYy1i+8eLH4l6tnMtFHB13Q5rc7XJcq4gY1yvSPVAID2SWo6GX3NSq\np8yc7P9GrJCFIsO23jy+K8ygTaWhiwZdWKT3svtexlfm7CscK9Cis7aFqWPbcd/St/zv+/IO2rJ2\nyEiJ5XOjYyjy0vcXAoMjd/8totCzTqOhW5KHHkgujpecK8jlwq/ppIMm4t6lbqpfuWoVHcf3iDft\nHMCKzT1Kg/753y+MbFPBPXQuTWUsfcpksX5lLPI9dL69v1BKbBT48xKR5U4RsT7y7y64LZByVLld\neKPbmSS5eKfiK5QVZYOeSaehc8RMinJ5kuyyw9Qx9Y2WXMpf0mSIcdVZswC4Sfs5thU/pTvvpWYF\nPP27TIMu75+zLX8fbuy57lwohVcNKglhiwNFB3/2FvIl0rfmJY2H3t1flGb2OXGrZ4VYsno7zr1l\nMV5e341vnTw9VHZ+jbJhSPI2Fq7YElrHUrxPs/YcE5IOSo7bYGzvKyBnW9hjbDs29+Sxbnsfuka0\noK9QQmvWDjXMYnlKDoNtRXN19BVKvkGPGp5wlzlrW2jRSi7haCVAllzcAemsoKEzAEftPRbHzwgM\numpQVFyarS9fQslxQhNyymGg6KDH89DdayTo0naK929iZyv+/sI6bOnJ++9Cf7GUuGxdvuRElpYT\n5U1ZchF7WKpxD9W6p3EeelhDd4/HB2FlDz2thi6XRY5y6S86oTu6d1cHXt8YjkRynCAPfWh7gyWX\npjfoHDkWNs5DLznMf3htOTtx4Vq55ZUlF9si31C8ubk3FHa2bN2OUCUXPfQtPXk/Rtstl6a8QkSI\nyLbesIf+vTui+WiSENfQFDX0D1/+OB791pzQvnH2fFtvHh/xpqFbFC3rQNEN/eM4zM29AgCHTh2N\nPTzp5OiLH8Qnj9oDA4US2nLhbIjiy1libh6Vdsm49BVKfjy27C2S5KHbVnkeem++5BsvnhZZnilq\nSb1DbtBveXoldvS5DYJo5HryRRRLDK0ZCz0plmeTGSg66MmXfCMWBw9x/cjhU/Dy+m4sWb0dq7as\nRodnxF3JJd4C5YtOxNDGrRwW0tAVkUmqhZyLJe/ZJoQt8tvMB5B7C0X0FxxBcqGyDPpOhUEGgN89\n8nrovK2KtVJ39BfQVyjhWycfENbWh2KUSzNAoYEZ/cQT3gXkFakjRfIeOd+EXElKTniW2Mm/fMQ3\ntPcuXY+f3/9ysK8UaigfJ267/LutveEws1Vb+rBqSx8+/PYpqQdHxUPK1yVHIMRVzn7hHokhhQdM\nHImpY9v8uGn/WMK15mwL08YHWvjfFq1FX6GEtqwtlS+YvOM4rvbeIsWR9wseulwHZA99oKiPCrIV\nHjogJOdy3AUugjVFmR+CKf7mn15I6rdvfR4/8sLgRIO+o89d0T5uxmocA4WSq6GnqMfc8/3MsdNC\n8zR4Q9JfKCUGC+SLUclFtVg7R6Whi4h1jAn3NqNpbFWvCHfI+rwMnbwHkTYOncOdPFmn/5/5r4Tq\noVzngGAd38mjwnMBKul1VcOwMehipVJNDOCM9lpvvuJLe4qFdGWJRf4szhQFvHDCGKOtWu6OQNrf\niPk6RLb25pUVJpeJj8MX4dOSHSc60032nuLqpjhBRDx3R4uNrG2FegJAuPufy1iYManT/7zb6FZf\ncumR1qAUc767g6bhl76/UEI2455f7uK7GrpbzoOnjMIhU0ZFJtf4+1rRawGCe8DDFv2Zot5nPiDL\nue3ZNTjtV4+GjiGWeXtfAaUUBl3noPQXHfTmS0E91jz2Py1cjb88s9o7loWDdhsV2ae/ENWKR7dn\ncemHD/Y/54tOxGjHeugKDV1ENOi8h1gsOa5Bz0XNkypSinvoA14d5M/dNejpp93f8dxa77jx+8kN\nGuDOUwGArhFhg97oJSCHj0GXKpnO0+DdMT4xo6OlfA9dllwKJRYKv+rI2Vr5xHH0D1nvoXu/lT30\nnnwoFpqTS7ESj18eBnz9lsX4+p8WR/JeiIOaqvOL9As9FNGQdrRklL0F0ZDmMhYsi3DsvuMAuC9M\nX9416Ht3deDovcfhlJnupBl/PU/uoUsedkEI3ZQnH1lE/ljHl+fsCyLCVk1IoJyTR6boSS5cRmCM\nh2BG5b4Vm8PZOcXZqdt6Cyg6jnbGqlwemXzRwZptfYn1uOQwPLDM1fVti/C23aMGfaDo5pMXY/93\nH90W0tXzJRbxeuMMumj8VBq6aNDzJQevbdyJDd0DyGhCSlUrdO30Gn1eB3mviTcmumRfG3b0J+ZH\nUqF6Em95kw67pBnqS1Y3ZoYoZ9gYdLm+6x4KTxvLK1JbCg9d1tDlCl0oOSGDPnVsu9YrkKcMc4iS\ncyqLp+3I2djaW1CGKmZtSzvxRobBNTgvrt0RqfgRDz2moovhZxaR36C0ZKyI0QUQCh/j4YVXn3UE\nTjpoIjZ2u6vSt2VttGRs3DTvKBy+51gAQe+IyxuqxiKrk1ysYICbl2nOdGXGiiBsUXMbg/QBnofu\n9XR4orE4Ih56iUUGI4mAk8WZn8IhVca/nIUcMhZh1p7R5Kh8gRAxXFSepJcvliLheXZI7ozz0FUN\nuzCIXnRw/M/+iTueW4uMRX6vRTxkKA2yV1+DGck8co176MF4h8jti9fgqzctwuwfz/cnmYlUkkjL\n99Alg37OzYvLPlY1DBuDrtM6ZUZ5Czv4kksK7VIerJIrSNFx0JcvYZKX+W50ezZWJy83Gb5qYtHE\nUa3Y2pNHyUHUoGcs5cvDmT1trP//O59bi8WrtmHTzgF/KjdHXohYN60eCGvoFgXeWMayfAM7sbMF\nX53rhi5u6w0845xn4FqzNvabMBKbdg5g50Ax5KHxl5OHtPEoF5UGzrfJkU4WkX8NfJ93HzABb1x8\nCvaUJihx2URnnHnDEF5TNKqhqxA12G1cQ5eko5m7j9I2yqrFSVSNpg7bIozpyGHJhSeGtvcVSl6j\nRH4D4xr0YB83Qkwy6HFx6KKHnnBfRKcoYxPGdrjXOVJorMRXROe02b5B5x56uLzn3LwYd3ryyvOr\nt0d+n6SSqDpLG7oHYFvkS7qDxfAx6NJd1q16wj30P3t6YpoVTeTBQe4J/OADbwMA5IvuoOjxMybg\nnfuNx79e3xJaI1HE0XropK1IqhXPJ3W2ehp6dKJH1rYik2o4//fJw3H0PuMi27f2FnC5EIUCRA26\naiYcR9TQHRbIHRmbfOM5dUw79hzX4Z+PI5Z/QmcLHAas2dYX8lqDlzPQ0N3JQ9wwd0WOJy8vR4h6\n6AC/9+qbHyd1AMF18pmihKhRkxHPvb3XNeiih33ee/bH7z87W2sAVYuT8ElZafpl3OCNbMmEGp/+\nguOPTfDGNGNRaEwqX3QwUHTwngODWbhxS0FmyzDob20P0vtmLDeVwZ+/cDQO2yPoTYhOj0665Ofk\nf+WxoY6cjUOmjMKMyZ2+Zy1SySzy3nwJHTk7MY6/3gwbgy6/d7oXVG5B0wweRg26W0HmHNCFPce1\n+x66KxHE31J5WS/5OxVi/vS9x3dgxSWnYkxHzu2us+ioe87We4kWqb1aAFi0chuO2jvw3nf0hz32\nuAWGRYPeXyiFBqb4+TpaMr63F/bQg7LylKn9BSc0UCiGCwLuPbEFDV28d/xFVk4s8g16uCHXvcNJ\nHnrW99ADDT0pN7947v5CCcVSOBTwbVNGYXR7TpvzvVPhBfJrTmOK+L0kolACLFdyce8Tv/eyh+6w\ncCQR34cjGzTxc5IMuFrI156x3YbkiGljQ9ckz7tQXp93npzkBHD6iw5mTRuLaePa8eaW3mjvPlFD\nV19HpZFKtWQYGfTwTZ48qk25X5p4XRnZU+VGxfY8RK6ht+XsiKGQ0WXlI6SYWOR5pQCfNOFGp8jn\nzNqWciSelzkupPEd+473tduyPHTB2ItT4m3Bi27P2f7Ls03joYv6sCi5BANcXEN3pYETZkzEQbt1\n4mvH7xfZV/YIiYJGSW7UKjXo/kxR77mm09CDc6/b3o8d/cVQT1EXdsnh+/LG9/+duD/OO3H/2HOK\niOVrDTUujj8O0Op76Fakgdo5UNJGr8geeng1o3hzw7OZyscUNW1ubDftHEC+qH5ovNfEo53kwf6S\n42bJ3GNsO1Zv6Ys8r6TFyHUMBYM+bCYWyRwydTTu+uo7cNqvF4S2y93VNN2rqEF3f8MH/3rzJTjM\nfaBJHrrDmHKpOKIYD50Fkgt/QTJeXhJVDvSsbcV46BSbHqCzLYuvzN0X9yx9KzIoqspdzeExwLy8\n/kslSC5tWds3DlsFD11sfMSZm2LYGj8eD0NzmOv9jWrP4u6vvRPbQ4OsXO6Jxkv7Grr0ne7eJ0ku\n/HkwL41DuRr6i94iz+J5uCEXPVrxiLyhO2yPMbh53tGhY/P9crZ+BrQY/SOWZaBQwsbuAYxqy/oN\nq21RZL3enoFiSNILDYpK1y7e5mTJJTDoolwmvqIOY1ixqQfvvuxh7XGyQu8QUOctytmEzrac8h4l\nzRzXdcCSIpUaweCXoI4ctFtnZNsoaS3NNHLZxp3h5df4wJxFrvfJR+pbs7Zy0oHIS291Jxpumd8/\n8aY7WYkFL0zGtlDwBlhVg6K6l8eyKLbrO7I141dMWWpKr6Ez36hlLAst3ovVlgsM+ra+cNgiR2wQ\nRe+Rv6SLVm7F129xIwfChiQoC5dw5MY1JLlIz0mvoSs3+8exLNeA81wuPBVwHKpenJj0KeN76KLO\nH+zLDbpqgWV/n5ixIXEQkzdsHTkb3QNFzH9pA2ZOGRWKFJIlhr5C2EMXjbhK5gquS31f+G/WCRq6\n+Hx4YqyMd59XbY0u0i7C759OQ+ff6ZyvgRjHBdDXlaHgoQ9rg66aYCRLLmmm5i6XVpLnHrorJ5Dv\nwfMwuzi+9ZclSg09zkNfuaUXL73V7cVeu9v4tGZHEbaYs/VG2yaKDBaKdLZm/WuQ49DjPHRRcnFY\nYIBFyaUta/sem5jvWyx/a8hDjw6KPrBsA/62eK137OD8ovHjRkp+wUiIQ5c9dJ1sGjRM4fsprv7O\nUx0wpPTQFYZEzN+uklxEw9jq3Rc+SUpF3OLX4nH9iCBh20G7jQolOFO1T1pJj2IMuua+8Hdyg5Bt\nU7xHvEPbmrXdpGwJQ788bFGnoQOu06O7hiQPXTcGpkoJ0GiGtUFXIQ+KppnI9Yq0aouvoRMhI3jo\nbTl9qy+iknkIycmZCk7g+WZty08QJZ/TlVzU5RBDClWMbM36x5Mll/U7BnD74jXK3/VLoZ2Bl0S+\nltmu8dDFF0t8KVqzUYMuSjWiZyheLt836qFD8NDlly9ecpkkTen2PXRvENSPcqH41M2qcs2Y3IlZ\nQigpfz52guRSsYeuMOjiAPiJB070G+SMrR7k1cl2UcklWUPnBp1nfwTCz547Xa1ZCyXBqdEhe+gq\nyUUeZ5q9V3D/48JzAX10TVLvvBEMfgnqzKUfPhj/OOedfpxxxENPYdFVU/0B14jk7CB1bpooF0Dd\nwsdlW+S42mUQDsg9dIsIN37uSH+/jG0p82a4ZSZt9AQAdLZlfGOnWk3mnJsXY9r5d+Ohl8ILVPVL\nL0GgA1u+rNWaszFjcieIwmsshgbpEgZFtwgzO+2Q9xf8jifnkl+wtdv6sNrrrqf10Pkz2X/iyND2\nfCmQXPhK91xDV3mi4ia55xBtkMn7jdpDP8Iz/sdLC3gAgTQT1/0X7xs/9zdPOgAXnX4Q3rj4FEwd\n2y5ECkUHRQH4Mlr02OHPoSgXTZ3kaXI37VR76Lz+tGRs93kkGPTA6VEPigJuL1ZslKaOCeYhxEmL\ngL4n3RSSCxFdQ0QbiOgFYduFRLRGWvBiSPLRI6ZixuRO3DLvaFxx5uGRm55kRFV1UJRcMjb5OZ1b\nsnao1X/nfuOx34QRkd/3F0rKZEpJeYR29hdDMdZFT0O3LcKRgochxxeHr4e0xh4Ie+iqXNWcW59d\nHfosL1Tghy1aQc+jPWtj6th2HL13OA4+nPgouC/idHZeJjFNctjgBcfI+R56+B5v7S3g+ifedMsl\nWR5dPeADdftKzzHw0MKl28QAAB41SURBVOGlvg2W05MN4E2fPwrnHB9EoUTCTBU9LCB8X8RDTp80\nEi//8L04+W3hNURF2mK8RSvkobv3aOrYdpx1zDRfpuSNYkYKW5TLKCPXO5sIf/z8kZh/3nHaniFf\nak6sb+Kz4xp6S9bSLiQSLlv4GlQSSk6SXMS6luSh6xJ+ybbl1i8eE1/QOpDGQ78OwMmK7b9gjB3q\n/ft7bYtVeyaNasVJB02KVM4kgy6/yIDgoROFKkVb1g55ei0ZWxlO1lcoRWKJCWopRs7H7b9oNvkx\n7RaFve7OtmyoXOJ0ZNuKRn+IdLZmlL0MeQKWbIRkfZ0bNdtreICgwu8+OhxSGk5NGhxXXIJMJQ2I\nhokE7ZrvG9dbksdXdD01PgCnN+jk9650US5dI3NS4xTvoaueD0neetJCJmmSzonnlr3nFt9DJ+VY\nVHrJBThmn/HYp2uE1smQ1+gVywUEvaeWjI2SE80lI8N7a6KGLk/nd+dHiD3D4JnEzbcA3KUPRfh1\ntUr3JE2eqFqTaNAZY48AiC7N3aRM6GzFJR+aic+9Yy/87yfenqihxxl0PijKacvaklFmSk+mN1/y\nJ9CIqIyKvKalvHhDvuhEZiaOasv63tDsvcbi1i8EngKResB0XEcOFgEduQxIkfSKpzXgRLIcarwa\n3vCIZVbNdOSIL5Y42UZlQOTL4Pcq8NDTK4q6dv2UmZMBuL0tkbwwjmJb5E39Z/76qCKyDCOXKyK5\nePsyQdcXbWpcODcfMEzb/ef3VdaFxSgXXvRxHUGEWNLSfX5ZNbKYSL/CgLYoNPSc7a7GlDRoqQpb\nlGWSrG0hZ0cnrgFhyeWwPUbjsjMOCf32fz52KC79SJCBkj/vSD6e1MvN1I5qNPSvENEST5KJZvrx\nIKJ5RLSQiBZu3KieDt9oPjZ7D3zntANxyszJvhH9yYdn4rPH7hXZd98u16CLy2H5kgtRqJK25+yQ\nUXYXQFAbdNlAOUytzYXWtBwo+l4Fr7Tb+goRb2xUW9Yv19j2XKii2RqN96xjpuGi0w/yPSzZyEzo\nDHsl8vd9efUiIVk7WBKNNyRxk7tEQyTOYlRNhop4fH4EENfQ1UbnwMnRcFZdT+39h+6ONy4+JTJR\njQ9IcolF9NBlySVjWf4CyYDKQ49ODJMRj5hmdnPSykP+ub1zyaF6fCDbEjx02yJMn+SOJfCGTkb2\n0MW7qiu3qvFReei5jBsIoJJEQovECwPyAJTrtsrpMcR3Qjx+1rIiS+GNG9GCj86a6n/mP5WvI8Vj\nqjmVGvTLAewD4FAA6wD8TLcjY+xKxtgsxtisrq4u3W6DBn+RR7fnsHdXR+T7MR05jO3IYXdh0KQo\nDIiJL9+YjlzIQxfzZYv05YuRyq2bQSp6POIkIm6wt/TkIxneRrVlfcNvW2EJwJIaIc7BU0bhzKOn\n+Z955RzXkcPHZ++Brx2/H37/2dn+97JBl/V2fim2ZflTtHk5onMBgusWyypKLipvWzYe/jqxGe6l\nRn/zqaP3xF1ffUdke1xPTSU5DJTCUS6+ho6ofGHbpAwV1H32e1BCmXQDpDriNHTVuWWvV+Wh2xbh\n+s/OxsP/790R2YEje+hilVblF/rmSQfggvdOj2wPjTNwDz3jzspWSS6iJMjvtRiHLr9buQz58iUQ\ndrxCDQaF66EK/jxkySWtrFdLKjLojLH1jLESY8wB8DsAs5N+M1Th9ziryX+SsQiXnXEwzj0hmFqe\nL4nhg+5fImBMey7ioWcVhqg3765o/8x3TvC3lRx10i7ZaGUl7wMAxo8IG8hcxvKNNlF4X8tSv1iy\nkeca/0dmTcHFH5qJY/YZj8OFlKuiEcp761qqU9mSH9XDXzQ5dFQnd4Q8dJVBl18Yv2uuHhR1t1nK\nsMJylwrjHi3PdRI3UzRjhQ16ZOBQ84xFvnDcPv7/0xn08iQX2evNKTR0iwgTO1sxbXzU8eHI16Jr\nrDlfnrOvn1URCGQd8dn5HrrtjseoPPRww2GFrq2gWB/XlVyC+yyWTVz7lhCuhyp4Iyb3COXHVEnS\nr3KpyKATkdjf+iCAF3T7DnX4i2xb6unylkWYO30iDp4SLAhQKDn+Q/R14bZsaBIN4FZm8UX+79MO\nBODmgG7JWBgneDkFx1E+cNnbCwZF1YOe/n7CCym+3LYmbJFJcdhf9AzInAOCfOGiFyQ2Pvt/5x94\nbvV2ZcXPWJa/LzdEsuSiq+aiV64yzrI3KHbN5d/HHQcApnlZIP/4uSPx239/u6ZEAdyj5Zq5n8vF\nUsdii+MW8n2Sa52caOv8907H2e8I5MA0XfnWlJILX+iCr+kql0H00GUDteDbc3DnV8K9nUivSXi4\nstPQIaTo9cvt1dWccL8cwUMHwr1Bvk1cFzgja+hFJ7JKmJg0jl+nitMOnowZkzvxyaP2UH4P6ENF\n5Ya3EcvRpQlbvAnAEwAOIKLVRHQ2gEuJ6HkiWgJgDoCv17mcdYPbUIsCAyE+hyB3SnCriiXH17Z5\n5eFexhfevY//wjosPCi6jzfAypdXEyk5TD0oKlU01SxCedkrQJipKUXA6OKk5bbkw4dPwRMXzMVR\nQoihnEYVQGiikSoLYMYm/NsRrt4402sU+f0Z2ZJBe87G+w/dLfI7+XxKD13zEsYNiuqiM2743JG4\n6lOzcMy+43HqwWp9WCQysYi5xku1YpHroQfnHdGSwXPfO9GfHSobSv7MuXdLiJ9eryKth37G4VNw\n11ffEYlp5/VWzB4pG6gpY9r9Z+qXTZZchOZaLHZLxsLtXzk2dC4geJ/EZ//zjx6K9x2yG2Z6jY+Y\nt3+alMceECUX92+hxDQautpD59xw9pH45FF7wrYIP/zAzMj3/nXxKBdJ5iKEe8NDwqAzxj7OGJvM\nGMsyxqYwxq5mjJ3JGJvJGDuYMXY6Y2xd3UtaJ0TvkXu/onHjDzq8+nyQJGuiF/3BK+CIlgy+63ni\njiMlntLkLAHcyUaiNPg/HzsUf/qPo7UrwIiGafzI6IIH/MWIpDOVQi3l6xTRZawEguXCxBVZxMEj\nMf/GiQdNwopLTvWPx8u+2+g2vPj9k/0c6XHIYwHuNvW+vocuGLXTD3EbDV3kS9fIFpxwYHSijg4e\nCcGTVzneikWu8Y2WXWxEMxZhVFvWL58uGoLXQ9ngp8m5nSbPv3ts9XJ0fByCT1wD0jUk8jMRbRgJ\nzsTZ79gL+04YGTku/794zQdMGolff/ww3wkSF5yZvddYfO34/fDNkw7wt6lyucjGNKfw0J/5zgnY\nS5CTRrRmYtcn5vsGGnrUQxdlnUasL1pNlMuwgHfniNSrFykNuhMslHvIlNEAgFVbgoRB4rJkYldb\nl7MEcBd0WCkcY8qYNszea2xEjlCtl8kHqH718cNwxZmHe2XwJJdIGFl06v8nj9oj5ImnIV90IjNe\nVYNHKnln7/EjMHlUKy44JToYFoes0eu0ZB7uKBpv/kInxW/HIdYBcWKRH+UC9aBzxrJC9yEjPRt+\nGR9++xTleWWDH6eh86/SLK0YB8/34/Y6wseOQy6bHP8drGQVbuDk86qukdfpyx8OFmIZ29GCb7xn\n/9CKU/KgqKuhS4O+mXDIsW27EqgYO56UHfKOrxyLBd+eExh0hYYu1rdGDIoOq/S5f/zckcpufxx+\nNAaR0qvxIzPasrjo9IPwvTuWuoNf3naurbeEWnvu6YdT2+ryfnPE2ZfcM5BH9MWp/xwuYXAvVNxP\n5S3KBuKC985I5X2J5ItOZMk6lYauivJpy9l44oLjyzof4L4c4jJlSQZdfCH5C52UPC2OjBXE1HOD\nTkSYOLIVj726GVt68iHpjhPx0L1nx3sxfPfLzjgYPxXim3Wvf1I2RyC95KKD128eW5/2vNFBUem4\nwoQzTkYhuaiqo27+BCA1CsL4kW3xFBnh3+WkQVFeT0UnJWmC0cjWLEa2ZoWwRTnKJdxDH7KDokOV\nY/Ydr+w+xsFvsmWRMnZXrKAfPnxKZHtHSwaXfuRg3PIfQV5qXrkYC1dC8YEnGRZe8eU8FLzFF70L\nXXSJeBzxuPJam7pp3Cqe/q8TMLo9i4GSE5nur5osVW5DAQD3nvsu3P/1d0W2y9617th8OTexu1wL\nD100GrxhsYnwndNm+HlIiKL5wzPSoKhYP0Tc3wb76SSXNJ5ytQbdj5xkzHcA0kTXROPQ1UZMdT/E\n/6tkKFWWUD52JTYQdsi4k6ehRx0j1SIdoiwyYWR0bEqFzkO3iPCTDwcN9JDQ0Ic7XHKxSD0ZIxRu\nJlRosdJ8dJabL8b/jWcgS9KgqDgzLWkmHz+8nPrTz4IXypUdrfwZTdfVsqK5XHTrj6roGtmCSZ2t\nbqiiZNCVHnoZjQXngEkjsZ+UDAtQ5zdXMUJRDv4ylTN7VCbIUx6c17LcBZ0neJFGypmikv7v9+D8\nLeU1eukmFlX3agceerQnEVu2SOx1+Hv+rUo3B4J7rDpXNhPdOM436OryZG0L+aJCQ5dyufAGhu/3\ngw+8DVPHRgdcVfhT/7PR+jln+gRc8qGZoWPXE2PQfS+I0J6NGgLRaIiVLG4Zt2Al+LBXIU6W4A//\n+s+qQ/j5eeUJHy2+hx7/6Lj3IXtIroYudw3LMygtGUspuagkq6R1JMshrYcupgzgFGvgofOGTzwG\nkbf2pZccTTVT1P2t0LBHJqCoz/eZY6dh3wkjcLoUBRQnffBvqs3NzesEnwELpB0UDe9z0O7hWbn8\nuDoNnf9edQ9VE+LGjoh66CLuUo3qOPTwed3fc09enh3KEdPscnhR5V43384bR2PQGwAftLFJLbmI\nBkmsrHGeXtCllgZFQxq6e67j9u+K5AkB9C+5amJRXBnkSXVuGGN1RjbnGXQuuRy3vzcDWFHoSiQX\n7Xlt2aCr9+uQDPr4ES1+T6waD92PwBC3edfc5Q1MW6QeTFPeByEsUcXUse144BvHRbr+adrfanNz\n8+K64ZjheQRx8F2O2nssXvrByZg+KZpmAQjfD9GheJf3Lhw8RRF5o6i3YxUaevg3liaXSzjpmP++\nlMKT00Re//EpuGXeUZHtcYOioWObQdH6I1ZWlYcpVmLRM4rz9DJ+hEBYcgl56MLvxQyCHF0om78A\ncoKHzs8rj6y7kkt1L3suY6G/EEguXGoRS8y9ukokFx2yMdb1LMRopaf+83i05mx8/vqFAKrV0N3f\nionIeP3w74Fipiig1n/5kymzg5SqR1Vtbm5+XXz9Vndbih8K1S2uDDoD/L5DdsO/H7lnaPZo8Jvo\nPRzjpZEY0x7dH3BlGl0cugh/ZgUpkZyILlyUb9Yl5+LHNoOiDeDfjnBngO0+pk05kBT2JILtcZ6e\nH7boBP//2tx9lR46AHQrFpLQ2dysQseNLQOTJZewh/7aj8tPZZ+zuYfuNkTcg1QZy6RylnVeOeeJ\n7gUTtk/obEVna9b3jnQZ/9KQtaOeFj8VN+h9+ZLS4KqM/Nzp7izcj8/Wz0IsF37uztYs/nB25Rk5\neHHFlBRpGhK/kUoYF9BJJFnbUhpzIJpG4wfvP8h/H1R5mPjxVHHossGWn205Db8/sUixQpb715Ov\njIdef848ak+cedSe2u/lrqFFbjc07oGLM/2ICCsuOTWyT8h7UdR9Xfc2l1JD9z10yaDL2RYrkUS4\n5NKTLyFrE75x4v7IZSx87Iip+Om9y0P7VmNAZWSNMm4cQ4Z7aNVIQKpeEX+ZeQy+OAVdRCUXTBnT\nrqwb1eBLiBbhnftVngwv0NDF9BgpDLomMsc/rvc3aVKYCnkwX5VMLnI823Kn/nvP/9cfPwxjO3LK\nFMdAkHivnJ5lXJQLEDgejZhYtMsbdB08P4cqptgpRRdmFlGtOCMjjohf9alZ+McLb4WMoe7dyaXV\n0CXJhRthssofBI2UIWNj+fpujGrPoj2XwYiWDM5/7/RIGCNQvpwQf17JqyrDi+JGqZoeg+q3fJ4A\n99B1Bj0Y7Kv49GWhKuuSC09M/XvbN+hBSoo0ZU+KiGnL2egeKMZ66DqSZEbd8e57cT2eWuEu6dA1\nskU5iY5HVVUyeB7EoZtB0SELX9BBHkDkxjAujjyjkTtExGnpe3eNwJfn7Ks8j0xaD52/0MFqL14j\nk7B8Vxp4Otyn3tgSGncQvZ56yIVtWTsx/l5HsVS9h66653yBcO6h93gG/Y2Lw1IW76l0VDmDMy0q\nvbezNaucK6D+vfvXYQzTJ3fCtghfnbtf/I8geOgaySVpEDM22CDBiXnkm3Nw+5ePDW3z1w3oLcSe\nl6+ixQ16OYPnbjqN6NiJn0deaBzrjfHQNUzobMGabX2ROHB5VRwVgTGNPsC2rI2+QimxwvDu2vzz\njgMB+PffPYm3dvRHZorqyiEPxFz76SNw3eMrElOBpuGV9Tv9/2/sDtb4rKVeruJLc/bBcft34Vu3\nLgEQ9aJ+96lZfpdZphzZQIfqt93eAuH8vnKDLjfI3EDKETi1RjYinAe+EZ2oFQdfNKWzNYtRbdnU\nYy1JA71umopu7XOIc1TEuv7Uf0VnGu8xrh17SMm6RkgNmHzeaz49KzSgKq+ulQaLSBkm6s+wtY3k\nMuhMGdOORSu3YauQGxkIKkRcl4y/VKoGefzIHFZt6VNqfifMmIgHlq13j+Ft28dbMWny6Fa8taNf\nkA7ipRdeTt5dnjVtLGZNi8bQVsIrGwKDLlbSNEmjquGg3UaFBszkl+49Mcm1vnb8fvjSjc9G0sSW\ng3yvO3I23u2lFx7pGepujeTCHQNVIrV6wJ/FHz9/JEa2ZP1EWGl5z4yJ+O5pB+JjR0xN3rkM+POT\nZ25y4hpc0UNPO4tTjieXjz93erjOVDYoGu5xE8HPjQ8IHrox6IPHhe87ECNbM5gzfUJoezCJIGZQ\nNKaLNa6jBau29EUSWwHAVWfNwnE/fQhvbu6NfHfkXuOwaOU2IZWpu12nI4tRCrXmijMPx23PrsYp\nMydrpRWe5ChN7HI5yNO603LKzMlVD0DyRvQ9B07Ew8s34IWLTvIbb+5592gM+rRx7fj6CfvjjFnq\nBFy1ht/2Y/aJznFIg2VRKAd7WuRkXDLcoG/emY/dT0UlA+xybqe070N5GjopczPxqmri0AeRx8+f\nCwZ33cAffzCaAzmNh27FGPRPHzMN596yGFPG6FPTAtHETP/vxP0xe68x/kw13iX+lCZCJyhD7Gkq\n4qSDJuGkgybF7vPrj78df164CjMml+cZJhGaC1DDGPc0cA/xk0ftid99alboO55uQGd0iAjnnJCs\nQVdLg8ZctXDPeaYmpxKfqr+lp3yDXk4DzpEXU4lLCV3puYhI2eP2c+AMJYNORNcAOA3ABsbY27xt\nYwHcAmAagBUAPsoY21q/YjaO3UbHP3BfQ4+VXNy/ql7lBw7bHacfsptWnuBbo2lHrVD3sC1n45Uf\nvVcfi93AgRgVk0a14qvH196AiQazmklClZ3bvadywjTA1Zr/65QZmDtjQuS7RvLjD83EJf94qWGD\nrzIH7taJO7/yDm1D/tEjpuKO59ZWFHtfySQ1Pgh8ysxJ+O2/vz11hFeLnX5ilk3hqDWC65D5GvoQ\nm1h0HYCTpW3nA5jPGNsPwHzv8y4BpTDo/AHqRuXjtGZff09RlqxtaSuo3UCvoJGIDnAtZ6GmgXth\nuhfz8+/a2x/zGCzed8hueOz8uTVNuVAuM6eM0oYYTuxsxf3fOC514iuRSlJWdLa5DRtj5YXrqhKB\n6ZAHRcU1WIHGvouJzThj7BEimiZtfj+Ad3v/vx7AwwC+XcNyDVl4PY0LW5w8qhVfnrMPPqRZsCCO\nwEOvoHACVkykTTMjeuiNNugXvf8gjBuRC62z2kj++Lkj8drGnck7DlMqSVnBw0nLfQ3KkfNOPGii\ncqyIb9qnawSu+8wRWhmqllTaL5vIl51jjK0josHtZzYQHhUXNyhKRPjmSeWtxhP8uLKfyTQy9rWR\nVDooWgsmjGyNXVuy3hyz73gcs29lg5zDgUo89LgQ4vhzpTfo8961T+jzpM5WrNnW538e1Zb1o6Hq\nTd2FNiKaB2AeAOyxR+1yVgwWPNyq/gNy1RnipCiXaz49a9B01mrQZekbiuRsqyG66a5CJT0yXl0a\n+RT+9IWj8dQbmxvegwQqnym6nogmA4D3d4NuR8bYlYyxWYyxWV1dleeWGCrw9JrVpifV8cXj3NZ+\nUsrReB3iAgUq5k6fiCPLXEd0KDCI0nDZLLnwRCy96KTBLsawoRIDydP3njIzPiqLM6a9vCUsVew+\nug0fPKwx4akylbpodwA4C8Al3t/ba1aiIU6hzh76GbOm4oxZ1U/m2N/LTfGZY6ZVfaxyGVnH2ZBD\n3SsXqTaF7a7EzfOO8tdp1VHJQO+08R146Qcnp34W95z7LqzeGp0H0iykCVu8Ce4A6HgiWg3ge3AN\n+Z+I6GwAKwGcUc9CDiX4hKBGh8yVy9iOXM0z+aXhyf88vurVcgy7HqqEWTr2Gq9OlaujnIZ1Ymcr\nJnamm4U6FEkT5fJxzVflL9s+DKjFUmbDmWZ+GQxDn9u+dAz2rCJ9w3Cn+UbFBpmWjIWBotPwWYoG\ngwF4+x5jBrsIQxpj0Mvkxs8dib8tXoMj92q+AUWDwTC8MQa9TGqZtdBgMBhqidENDAaDYZhgDLrB\nYDAME4xBNxgMhmGCMegGg8EwTDAG3WAwGIYJJsrF0HTMOaAL+04Y3LzjBsNQxBh0Q9Nx7WdmD3YR\nDIYhiZFcDAaDYZhgDLrBYDAME4xBNxgMhmGCMegGg8EwTDAG3WAwGIYJxqAbDAbDMKGqsEUiWgGg\nG0AJQJExNqsWhTIYDAZD+dQiDn0OY2xTDY5jMBgMhiowkovBYDAME6r10BmA+4iIAbiCMXalvAMR\nzQMwz/u4k4iWV3iu8QB2tZ6AueZdA3PNuwbVXPOeaXYixliFxweIaDfG2FoimgDgfgBfZYw9UvEB\n48+1cFfT6M017xqYa941aMQ1VyW5MMbWen83APgrAJNkw2AwGAaJig06EXUQ0Uj+fwAnAnihVgUz\nGAwGQ3lUo6FPBPBXIuLH+SNj7J6alEpNRJ/fBTDXvGtgrnnXoO7XXJWGbjAYDIahgwlbNBgMhmGC\nMegGg8EwTGgKg05EJxPRciJ6lYjOH+zy1AoiuoaINhDRC8K2sUR0PxG94v0d420nIvqVdw+WENHb\nB6/klUFEU4noISJaRkRLiegcb/uwvWYAIKJWInqKiJ7zrvsib/teRPSkd923EFHO297ifX7V+37a\nYJa/UojIJqJFRHSX93lYXy/gpkMhoueJaDERLfS2Nax+D3mDTkQ2gN8CeC+AAwF8nIgOHNxS1Yzr\nAJwsbTsfwHzG2H4A5nufAff69/P+zQNweYPKWEuKAM5jjM0AcBSAL3vPcjhfMwAMAJjLGDsEwKEA\nTiaiowD8BMAvvOveCuBsb/+zAWxljO0L4Bfefs3IOQCWCZ+H+/Vy5jDGDhVizhtXvxljQ/ofgKMB\n3Ct8vgDABYNdrhpe3zQALwiflwOY7P1/MoDl3v+vAPBx1X7N+g/A7QDes4tdczuAZwEcCXfWYMbb\n7tdzAPcCONr7f8bbjwa77GVe5xTPeM0FcBcAGs7XK1z3CgDjpW0Nq99D3kMHsDuAVcLn1d624cpE\nxtg6APD+TvC2D6v74HWrDwPwJHaBa/bkh8UANsCdVf0agG2MsaK3i3ht/nV7328HMK6xJa6aXwL4\nFgDH+zwOw/t6OTwdyjNe2hOggfW7FtkW6w0ptu2KsZbD5j4Q0QgAtwI4lzG2w5vLoNxVsa0pr5kx\nVgJwKBGNhjureoZqN+9vU183EZ0GYANj7BkiejffrNh1WFyvxLFMSIdCRC/F7Fvz624GD301gKnC\n5ykA1g5SWRrBeiKaDADe3w3e9mFxH4goC9eY38gYu83bPKyvWYQxtg3Aw3DHEEYTEXeqxGvzr9v7\nfhSALY0taVUcC+B0b72Em+HKLr/E8L1eH6ZOh9Kw+t0MBv1pAPt5I+Q5AB8DcMcgl6me3AHgLO//\nZ8HVmfn2T3kj40cB2M67cc0Cua741QCWMcZ+Lnw1bK8ZAIioy/PMQURtAE6AO1j4EICPeLvJ183v\nx0cAPMg8kbUZYIxdwBibwhibBvd9fZAx9gkM0+vlkD4dSuPq92APIqQcaDgFwMtwdcf/Guzy1PC6\nbgKwDkABbmt9NlztcD6AV7y/Y719CW60z2sAngcwa7DLX8H1vgNul3IJgMXev1OG8zV713EwgEXe\ndb8A4L+97XsDeArAqwD+DKDF297qfX7V+37vwb6GKq793QDu2hWu17u+57x/S7mtamT9NlP/DQaD\nYZjQDJKLwWAwGFJgDLrBYDAME4xBNxgMhmGCMegGg8EwTDAG3WAwGIYJxqAbDAbDMMEYdIPBYBgm\n/H9pwvAxmkOUbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses)\n",
    "plt.ylim(5,50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用autograd实现的线性回归最大的不同点就在于autograd不需要计算反向传播，可以自动计算微分。这点不单是在深度学习，在许多机器学习的问题中都很有用。另外需要注意的是在每次反向传播之前要记得先把梯度清零。\n",
    "\n",
    "本章主要介绍了PyTorch中两个基础底层的数据结构：Tensor和autograd中的Variable。Tensor是一个类似Numpy数组的高效多维数值运算数据结构，有着和Numpy相类似的接口，并提供简单易用的GPU加速。Variable是autograd封装了Tensor并提供自动求导技术的，具有和Tensor几乎一样的接口。`autograd`是PyTorch的自动微分引擎，采用动态计算图技术，能够快速高效的计算导数。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
